+ All Categories
Home > Documents > IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39...

IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39...

Date post: 10-Jun-2020
Category:
Upload: others
View: 7 times
Download: 0 times
Share this document with a friend
90
Operations Analytics Predictive Insights - Mediation Pack for AppDynamics, Dynatrace, DCRum, Aternity and Splunk IBM Operations Analytics Predictive Insights Mediation Pack Group 2 Version 2.7 Document Revision: 20190612
Transcript
Page 1: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

Operations Analytics Predictive Insights - Mediation Pack for AppDynamics Dynatrace DCRum Aternity and Splunk

IBM Operations Analytics Predictive Insights

Mediation Pack Group 2

Version 27

Document Revision 20190612

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

Edition notice

This edition applies to IBM Operations Analytics Predictive Insights 136 and to all subsequent releases and

modifications until otherwise indicated in new editions Please refer to the license files in the delivered

mediation pack for terms of use Also reference the IBM Operations Analytics Technical Community

(httpsdeveloperibmcomitoa) for additional information and forums on Predictive Insights

This mediation pack was provided in cooperation with Streamweaver Inc

See httpwwwstreamweavercomsolutionsopstream for additional information

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

iii

CONTENTS

Table of Contents

MEDIATION PACK GROUP 2 I

CONTENTS III

OVERVIEW 7

PREREQUISITE INFORMATION 9

SWP MEDIATORS OVERVIEW 11

KEY STEPS TO SETTING UP MEDIATORS 11

INSTALLING AND CONFIGURING 12

ZIP FILE CONTENTS 12 INSTALL PROCEDURE 12 NEW INSTALLATION 12 NOTES 13 STARTING SWP SYSTEM 15 STOPPING SWP SYSTEM 15 UPGRADE INSTALLATION 16 DEPLOYED DIRECTORY CONTENTS 16 ACCESSING THE MEDIATION SYSTEM 18

APPDYNAMICS MEDIATOR 20

OVERVIEW 20 MEDIATOR CREATION AND CONFIGURATION 20 SELECT SOURCE 20 CONNECT TO SOURCE 20 SELECT DESTINATION 21 CONNECT TO DESTINATION 21 SELECT DATA TYPES 21 CONFIGURING COLLECTORS 22 CONFIGURING DISTRIBUTORS 23 SAVE STREAM 23

AWS CLOUDWATCH MEDIATOR 24

OVERVIEW 24 EXTRACTION APPROACH 24 PRE-REQUISITES 24 MEDIATOR CREATION AND CONFIGURATION 26 CREATE NEW SOURCE 26

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

iv

CONNECT TO SOURCE 26 SELECT DESTINATION 27 CONNECT TO DESTINATION 27 SELECT DATA TYPES 27 CONFIGURING COLLECTORS 27 CONFIGURE DISTRIBUTORS 29 CONFIGURE STREAM 29

CA APM (INTROSCOPE) MEDIATOR 30

OVERVIEW 30 NOTE 30 MEDIATOR CREATION AND CONFIGURATION 30 SELECT SOURCE 30 CONNECT TO SOURCE 31 SELECT DESTINATION 32 CONNECT TO DESTINATION 32 SELECT DATA TYPES 32 CONFIGURING COLLECTORS 33 CONFIGURING DISTRIBUTORS 35 CONFIGURE STREAM 35

DYNATRACE MANAGED 36

OVERVIEW 36 EXTRACTION APPROACH 36 PRE-REQUISITES 36 MEDIATOR CREATION AND CONFIGURATION 36 CREATE NEW ORIGIN 36 CONNECT TO SOURCE 36 SELECT DESTINATION 37 CONNECT TO DESTINATION 37 SELECT DATA TYPES 38 CONFIGURING COLLECTORS 38 CONFIGURE DISTRIBUTORS 39 CONFIGURE STREAM 40

MF NNMI MEDIATOR 41

OVERVIEW 41 EXTRACTION APPROACH 41 PRE-REQUISITES 41 MEDIATOR CREATION AND CONFIGURATION 43 CREATE NEW SOURCE 43 CONNECT TO SOURCE 43 SELECT DESTINATION 44 CONNECT TO DESTINATION 44 SELECT DATA TYPES 44 CONFIGURING COLLECTORS 45

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

v

CONFIGURE DISTRIBUTORS 46 CONFIGURE STREAM 47

ORACLE OEM MEDIATOR 48

OVERVIEW 48 MEDIATOR CREATION AND CONFIGURATION 48 SELECT SOURCE 48 CONNECT TO SOURCE 48 SELECT DESTINATION 49 CONNECT TO DESTINATION 49 SELECT DATA TYPES 50 CONFIGURING COLLECTORS 50 CONFIGURING DISTRIBUTORS 51 SAVE STREAM 51

PIVOTAL CLOUD FOUNDRY MEDIATOR 52

OVERVIEW 52 EXTRACTION APPROACH 52 PRE-REQUISITES 52 OVERALL SETUP 53 MEDIATOR CREATION AND CONFIGURATION 54 CREATE NEW SOURCE 54 CONNECT TO SOURCE 54 SELECT DESTINATION 55 CONNECT TO DESTINATION 56 SELECT DATA TYPES 56 CONFIGURING COLLECTORS 56 CONFIGURE DISTRIBUTORS 58 CONFIGURE STREAM 59 NOZZLE SETUP 61 OPERATION 61

SAP HANA MEDIATOR 63

OVERVIEW 63 EXTRACTION APPROACH 63 SAP HANA PORT DEFAULTS 63 PRE-REQUISITES 63 MEDIATOR CREATION AND CONFIGURATION 63 CREATE NEW SOURCE 63 CONNECT TO SOURCE 64 SELECT DESTINATION 65 CONNECT TO DESTINATION 65 SELECT DATA TYPES 65 CONFIGURING COLLECTORS 66 CONFIGURE DISTRIBUTORS 67 CONFIGURE STREAM 68

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

vi

OPERATIONS 70

STARTING METRIC EXTRACTION 70 STOPPING METRIC EXTRACTION 70 RUNNING SINGLE EXTRACTIONS 70 EDITING AN EXISTING STREAM 72

PREDICTIVE INSIGHTS CSV OUTPUT 74

FILE DETAILS 74 TIMESTAMPS 76

PREDICTIVE INSIGHTS lsquoMODELrsquo FILES 77

GROUP INFORMATION 77 OBTAINING METRIC NAMES FOR PI MEDIATION TOOL 78

PREDICTIVE INSIGHTS REST OUTPUT 80

LOGS 81 CHANGING LOG LEVELS 81

SUPPORT 83

APPENDIX 1 SAP HANA METRICS 84

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

7

Overview

IBM Operations Analytics Predictive Insights analyzes data and learns the normal behavior of a

system through KPI metric data streams This includes data from various sources including

metrics gathered by industry performance and network monitoring solutions A mediation pack

helps users quickly integrate the KPI data streams by providing a default model to integrate a

particular performance manager

IBM Operations Analytics Predictive Insights Mediation Pack Group 2 known in the rest of this

document as the lsquoMediation Packrsquo provides a pre-built model as a starting point to facilitate data

ingestion Predictive Insights creates a performance model that it uses to detect or forecast

behavior outside the modeled range including generating an alarm when an anomalous behavior

occurs Doing so enables organizations to pro-actively avoid system and solution outages before

users are impacted

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

8

The Mediation Pack provides the capability to extract performance data and metrics from the

following sources

bull SAP Hana DB

bull CA APM (Introscope)

bull MicroFocus NNMi

bull AWS Cloudwatch

bull Pivotal Cloud Foundry

bull Dynatrace Managed

bull AppDynamics

bull Oracle OEM

and deliver that data to IBM Operations Analytics Predictive Insights to perform analysis This

document will cover the installation and setup of Mediation Pack for each of the supported source-

data technologies

The figure shows an overall view of the mediation solution in context

Metrics are extracted by Collectors flow through to CSV Distributors which produce CSV files

The CSV files can then be made available to IBM PI for ingestion in the normal way Alternatively

any Collector can be routed to a PI REST Distributor which will deliver data to PI via its REST

interface

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

9

Prerequisite Information

Item Description

Name Mediation Pack Group 2

Predictive Insights supported

versions

v136 and higher Note IBM PI continues to support the CSV

format for ingestion and so earlier versions of PI ie 133 will

also work)

Predictive Insights Information IBM Operations Analytics (demo trial resources)

httpswwwibmcomus-enmarketplaceit-operations-

analytics

IBM Operations Analytics Community (docs videos

mediation packs) httpsdeveloperibmcomitoa

IBM Operations Analytics Predictive Insights

documentation

httpswwwibmcomsupportknowledgecenterSSJQQ3

Solution Name supported

versions

Min version Max Version

CA APM (Introscope) 9x 107

SAP Hana 10 10

AWS Cloudwatch EC2 REST

API 2016-11-

15

Cloudwatch

111

Same as Min

version

MF NNMi 1040x 1040x

Pivotal Cloud Foundry 233 233

Dynatrace Managed Dec 2018 Jun 2019

AppDynamics 41x 459

Oracle OEM 13 13

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

10

Solution Name Information IBM Operations Analytics Predictive Insights Mediation Pack

Group 2

Filename mediation_pack_group2-27-ibmpi-ltbuildNumbergtzip

Supported Operating Systems Linux

RHEL 6x 7x 3264bit

CentOS 6x7x 3264bit

Java Java18+ is required There is a version included in the release

package Note Mediators do NOT run on IBM JVM (the one

delivered with PI at time of writing)

Web Browsers A modern HTML5CSS3Javascript browser Versions of

Chrome Firefox or Safari since 2012 should work Previous

browser versions may not work well or at all

Note Some versions of Internet Explorer (ie 9 or earlier)

have exhibited issues Please consider a more modern

browser to ensure full functionality of the product

Ports Port 7457 (default but configurable) is required by

Mediators

Outbound ports per Mediator Type

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

11

SWP Mediators Overview

The metric data flow is generally from a collection component (a lsquoCollectorrsquo aka lsquoMediatorrsquo)

specific to the data source to an output component (a lsquoDistributorrsquo) Two distributors types are

support 1 CSV - writes metrics to standard IBM Operations Analytics Predictive Insights CSV files

2 REST - posts metric data (as JSON) to the IBM Operations Analytics Predictive Insights REST interface

Whilst each Mediator requires configuration specific to its source all the data flows through to a

corresponding Predictive Insights distributor The configurations approach for each of the

sources will be described in this document

The Mediators as part of the Mediation pack usually will be installed on the same host as

Predictive Insights This makes it easier to deliver the data from the Mediators to Predictive

Insights If the Mediation pack is installed on a different host then mechanisms must be put in

place to ensure that Predictive Insights can get at that CSV data eg shared disk drivesNFS

FTP-type jobs run on a schedule to retrieve files How to establish such mechanisms is beyond

the scope of this document In the case of REST distribution SWP does not have to be co-

located with Predictive Insights

Key steps to setting up Mediators

bull Installation

bull Configure one or more streams which extract the data from the supported sources

bull Configure the lsquoPredictive Insightsrsquo distributor stage for each stream choosing

between CSV and REST output

bull Start the desired metric extraction lsquotasksrsquo for each of the data sources This

should result in Predictive Insights CSV files being produced or JSON data

being posted to the IBM PI REST interface

bull Note Setting upconfiguring IBM PI REST interface is beyond the

scope of this document For details on this please refer to the

corresponding IBM PI documentation

bull In the case of CSV integration configure Predictive Insights for ingestion

using its lsquoMediation Toolrsquo in the normal way Sample Predictive Insights

lsquomodelrsquo files are provided as a starting point The lsquoMediation Toolrsquo is not

used for PI REST integration

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

12

Installing and Configuring

Zip file contents

The file mediation_pack_group2_v27-ibmpi- ltbuildgtzip contains the following

bull streamweaver-ibmpi-27ibmpiltbuildgt-1noarchrpm contains the main SWPMediators executable and associated collateral

bull docs containing the User Guides

o for lsquostreamweaverrsquo (this document)

o for Spl_AppD_Dyna_Ater_MedPack_v15_Revltrevisiongt

bull license containing license files

bull Spl_AppD_Dyna_Ater_MedPack_v15ltbuildgtrpm release with support for

o Dynatrace

o DCRum

o Aternity

o Splunk

o ExtraHop

o CA UIM

o MF APM

o NetIQ AppManager

o Note this package also contains support for AppDynamics Oracle OEM and CA APMIntroscope - these have been deprecated in 1x replaced by the Mediators in the main streamweaver 27 version

Install procedure

New Installation

The following section provides information for installing and configuring the Mediation Pack on

your IBM Operations Analytics Predictive Insights system

Normal lsquorpmrsquo conventions are followed

1 Verify Hardware amp Software Requirements

2 Verify user account privilege requirements

i The installation should be performed as the lsquorootrsquo user or user which can sudo ltgt

with root privileges This is required to create the Linux service

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

13

ii The installer will install as scadmindb2iadm (to match PI)

3 unzip mediation pack file

unzip mediation_pack_group2_v27-ibmpi-ltbuildgtzip

4 Install the bundled JVM

rpm -i --nodeps opstream-jre-18ibmpiltbuildgt-1x86_64rpm

5 Install SWP Mediators Locate the rpm file in the expanded zip directory

rpm -i streamweaver-ibmpi-27ibmpiltbuildgt-

1noarchrpm

After this you should see an indication of successful installation eg

SWP Mediators are installed by default to

optIBMstreamweaver-ibmpi

6 If you used root account exit and return to lsquoscadminrsquo user

Notes

Installing to an alternative location

You can install to a location other than optIBM as follows

sudo rpm -i --relocate optIBM=ltalternativepathgt streamweaver-

ibmpi-27ibmpiltbuildgt-1noarchrpm

Running as non-`scadmin` user

As part of the install if not already present in the system a `scadmin` account and lsquodb2iadm1` group are created for consistency with the default IBM PI installation and related file-ownerships

If you wish to run as a non-`scadmin` user

Change ownership of all the files from down eg

chown -R differentuserdifferentgroup optIBMstreamweaver-ibmpi

Edit the initd startup script

etcinitdstreamweaver-api

Locate and edit the line specifying the DAEMON_USER

[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=scadmin

Change the default `scadmin` to be your desired user eg

[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=differentuser

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

14

Co-existence of SWP-based and Accelerator-based mediators

If you are using the earlier Accelerator-based version along with this SWP-based version

note that they use the same default port setting of 7457 In this situation you should change the

default for SWP to something else You can do this by locating and editing the installed config

file in

optIBMstreamweaver-ibmpi confswpconf

and locate uncomment and change the line with port = 7457 to some other port

Settings for the Core Services services

Settings for the UI Service

ui

How the UI Services binds to a socket

connection

Hostname or IP Address to listen on The default

0000 causes

the REST API to listen on all interfaces

host = 0000

The port number to listen on

port = 7457

to some other free port

Also By default the UI listens on 7457 and the REST API listens on 7456 The UI assumes that the REST port is 1 less than the UI port Having made the above config change the following config entries are required and must be explicitly added to swpconf at the end of the file

httpport = ltrest-port-numbergt must be one less than UI port this defines the

port that REST API listens on lagomservicesswpallinone = httplocalhostltrest-port-

number server-side clients of REST service use this to locate it NOTE quotes are important here

Data directories

If you wish to relocate the various working directories (eg logs data) edit the swpconf file and modify the relevant entries in the ldquopathsrsquo section

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

15

Installation users

SWPMediators is installed by root as usergroup scadmindb2iadm1 to align the default PI usergroups

Starting SWP system

SWP contains Kafka Cassandra and the core They are all run under the lsquoswprsquo service

bull As lsquorootrsquo execute

service streamweaver-ibmpi start

In the usual way running lsquostatusrsquo on the service should give an indication that the main

SWP demon is operational

service streamweaver-ibmpi status

Once the SWP demon is running you use the SWP command line utility to monitor the

status of the child processes

Use lsquoswprsquo to observe the status of the system

bull optIBMstreamweaver-ibmpibinswp status

You should see a response similar to the following

In normal operation the five processes should be in Running state

You can also run the status command in a loop (like Linux `watch`) eg

bull optIBMstreamweaver-ibmpibinswp status -w 3

Stopping SWP system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

16

service streamweaver-ibmpi stop

Upgrade installation

service streamweaver-ibmpi stop

rpm -e streamweaver-ibmpi

Install new version as described above

Deployed directory contents

The main directories are created as part of the installation

optIBMstreamweaver-ibmpi

jre

The bundled JRE

bin

lsquoswpdrsquo all-in-one executable and lsquoswprsquo command line

content-packs

lsquoPI mediation lsquopa modelrsquo files and Pivotal CF lsquonozzlersquo components

lib

Library files (jars)

logs

Log files useful for monitoring and debugging system behavior

conf

Configuration files to control SWP

services

Cassandra Kafka and swpallinone (core SWP services)

Configuration files to control streamweaver

dataswp

Extracted metric data is stored under here

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

17

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

18

Accessing the Mediation system

Once installed you can interact with the Mediation system via web-browser By default the

Mediation system listens on port 7457 so for example point your web-browser to

httpltservergt7457

Weaver Pressing lsquoWEAVERrsquo brings you back to the above screen You select the lsquo+rsquo and a list

of source options will appear Any 3rd party tool listed can be selected to begin the configuration process In a similar fashion Destinations can also be configured by selecting the plus icon in the Destinations section

Dashboard Pressing lsquoDASHBOARDrsquo while cause latency measurements (the duration of last collection for each destination mediator) for each stream to be presented

Configuration Pressing lsquoDASHBOARDrsquo shows configuration information for the platform

including particularly log levels

System Logs Pressing lsquoSYSTEM LOGSrsquo presents the latest log entries (the full set of logs and be viewed in the relevant log directory which is described later in this document)

In the subsequent Mediator-specific sections we will give specific examples of

Mediator-gtDistributor Each example will specify a specific kind of distributor ie CSV or REST

You are free to use either distributor in each case

The initial step in each configuration case is to press lsquo+rsquo on the SOURCES and then you should

be presented with a list of available Mediator source types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

19

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

20

AppDynamics Mediator

Overview

Extraction Approach

Metric data is extracted using the AppDynamics REST API

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

AppD Port Defaults

REST 443

Mediator creation and configuration

To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your AppDynamics source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

21

Select Destination

Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single

file stream emerges from each Destination and we donrsquot generally want to mix data from

different Collector instances

Give this destination a unique name This name will also be used as a prefix for the

AppDynamics CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

22

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from AppDynamics is based upon selecting Applications Tiers and metric

paths SWP will extract all metrics for the supplied paths

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

23

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)

when data for the next time interval arrives

You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open

for the specified number of periods (as defined by the Maxium Time Window Per File) This is

useful when you want to wait for late data The trade-off is that the files are only made available

to PI after that lag period - thereby increasing end to end latency

For example if you had 5min files and set the max lag to 2 yoursquod have current file open and

two older files leading to up to 15 min latency

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

24

AWS Cloudwatch Mediator

Overview

This mediator extracts Metric data from AWS Cloudwatch system

Extraction Approach

Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference

Pre-requisites

You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users

The Endpoint regions are shown in the following table For the subsequent SWP configuration

you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account

has access to metric data

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

25

T

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

26

Mediator creation and configuration

To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the Source column to add a new origin

Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for AWS Cloudwatch account

Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least

one Region Multiple regions can be entered by separating each with a comma

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the AWS

CloudWatch environment)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

27

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

28

Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics

in those categories SWP will extract all metrics for those selected metric tables For IBM PI

only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are

selected

The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide

convenient resource groupings If you want to scope your metric extraction to particular tagged

instances specify those tags here as comma separated tagname=tagvalue pairs

For more details on AWS tagging see the AWS documentation

The optimum settings for the other configuration values depend on your AWS accountdata

availability etc The defaults should be sufficient to get you operational

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

29

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

30

CA APM (Introscope) Mediator

Overview

CA APM was formerly known as Wily Introscope

Metric extraction from CA APM is based upon configuring the system to extract data for specific

Hosts Metric data will be collected for all Agents and Processes on each selected host

Extraction Approach

Metric data is extracted using a JDBC interface

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

CA APM Port Defaults

JDBC 5001

Note

The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used

Mediator creation and configuration

To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

and then press lsquo+rsquo on the Sources column to add a new sources

Provide a name for the new source instance

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

31

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your CA APM DB

The Metadata Window Width and Latency control time window of metric data which is

extracted to determine the available monitored hosts for selection

Note It is also possible to configure selection to use regexp expressions which are used directly

in the JDBC queries sent to CA APM Depending on whether or not you check the regexp

button you will be presented with a list of Agents(Hosts) to select from or be required to

provide regexps for Agents and Metric selection - see next section

The TimeZone is that of the CA APM server This must be configured to ensure the metric data

is correctly mapped to UTC

Enter the connection information for the CA APM Host

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the CA

APM host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

32

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name

Give this destination a unique name This name will also be used as a prefix for the caapm CSV

files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

33

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from CA APM is based upon selecting the hosts and SWP will extract all

metrics for that host

Note If you had chosen to configure via regexp the list of hosts panel will be empty

You must provide a set of comma separated regexp pairs - one pair per line in the RegExp

panel eg

User

These are regexps for CA APM Agents and Metrics and are used directly in the queries eg

select from metric_data where agent= and metric=User

See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

34

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Note Sometimes depending on the responsiveness of the CA-APM system it may take a few

seconds for the list of hosts to appear Clicking in the panel often helps with refresh

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

35

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

36

Dynatrace Managed

Overview

This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api

Extraction Approach

Pre-requisites

The hostport and REST api connection token for your Dynatrace Managed system (see your

Admin andor Dynatrace API - Authentication)

Mediator creation and configuration

To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new origin

Press lsquo+rsquo on the SOURCES column to add a new source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

37

Now provide the connectivity information for your Dynatrace Managed system

Enter your Dynatrace Managed credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

Dynatrace Managed environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to produce IBM PI CSV files

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

38

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Select the desired Dynatrace Managed metric sources

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

39

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 2: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

Edition notice

This edition applies to IBM Operations Analytics Predictive Insights 136 and to all subsequent releases and

modifications until otherwise indicated in new editions Please refer to the license files in the delivered

mediation pack for terms of use Also reference the IBM Operations Analytics Technical Community

(httpsdeveloperibmcomitoa) for additional information and forums on Predictive Insights

This mediation pack was provided in cooperation with Streamweaver Inc

See httpwwwstreamweavercomsolutionsopstream for additional information

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

iii

CONTENTS

Table of Contents

MEDIATION PACK GROUP 2 I

CONTENTS III

OVERVIEW 7

PREREQUISITE INFORMATION 9

SWP MEDIATORS OVERVIEW 11

KEY STEPS TO SETTING UP MEDIATORS 11

INSTALLING AND CONFIGURING 12

ZIP FILE CONTENTS 12 INSTALL PROCEDURE 12 NEW INSTALLATION 12 NOTES 13 STARTING SWP SYSTEM 15 STOPPING SWP SYSTEM 15 UPGRADE INSTALLATION 16 DEPLOYED DIRECTORY CONTENTS 16 ACCESSING THE MEDIATION SYSTEM 18

APPDYNAMICS MEDIATOR 20

OVERVIEW 20 MEDIATOR CREATION AND CONFIGURATION 20 SELECT SOURCE 20 CONNECT TO SOURCE 20 SELECT DESTINATION 21 CONNECT TO DESTINATION 21 SELECT DATA TYPES 21 CONFIGURING COLLECTORS 22 CONFIGURING DISTRIBUTORS 23 SAVE STREAM 23

AWS CLOUDWATCH MEDIATOR 24

OVERVIEW 24 EXTRACTION APPROACH 24 PRE-REQUISITES 24 MEDIATOR CREATION AND CONFIGURATION 26 CREATE NEW SOURCE 26

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

iv

CONNECT TO SOURCE 26 SELECT DESTINATION 27 CONNECT TO DESTINATION 27 SELECT DATA TYPES 27 CONFIGURING COLLECTORS 27 CONFIGURE DISTRIBUTORS 29 CONFIGURE STREAM 29

CA APM (INTROSCOPE) MEDIATOR 30

OVERVIEW 30 NOTE 30 MEDIATOR CREATION AND CONFIGURATION 30 SELECT SOURCE 30 CONNECT TO SOURCE 31 SELECT DESTINATION 32 CONNECT TO DESTINATION 32 SELECT DATA TYPES 32 CONFIGURING COLLECTORS 33 CONFIGURING DISTRIBUTORS 35 CONFIGURE STREAM 35

DYNATRACE MANAGED 36

OVERVIEW 36 EXTRACTION APPROACH 36 PRE-REQUISITES 36 MEDIATOR CREATION AND CONFIGURATION 36 CREATE NEW ORIGIN 36 CONNECT TO SOURCE 36 SELECT DESTINATION 37 CONNECT TO DESTINATION 37 SELECT DATA TYPES 38 CONFIGURING COLLECTORS 38 CONFIGURE DISTRIBUTORS 39 CONFIGURE STREAM 40

MF NNMI MEDIATOR 41

OVERVIEW 41 EXTRACTION APPROACH 41 PRE-REQUISITES 41 MEDIATOR CREATION AND CONFIGURATION 43 CREATE NEW SOURCE 43 CONNECT TO SOURCE 43 SELECT DESTINATION 44 CONNECT TO DESTINATION 44 SELECT DATA TYPES 44 CONFIGURING COLLECTORS 45

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

v

CONFIGURE DISTRIBUTORS 46 CONFIGURE STREAM 47

ORACLE OEM MEDIATOR 48

OVERVIEW 48 MEDIATOR CREATION AND CONFIGURATION 48 SELECT SOURCE 48 CONNECT TO SOURCE 48 SELECT DESTINATION 49 CONNECT TO DESTINATION 49 SELECT DATA TYPES 50 CONFIGURING COLLECTORS 50 CONFIGURING DISTRIBUTORS 51 SAVE STREAM 51

PIVOTAL CLOUD FOUNDRY MEDIATOR 52

OVERVIEW 52 EXTRACTION APPROACH 52 PRE-REQUISITES 52 OVERALL SETUP 53 MEDIATOR CREATION AND CONFIGURATION 54 CREATE NEW SOURCE 54 CONNECT TO SOURCE 54 SELECT DESTINATION 55 CONNECT TO DESTINATION 56 SELECT DATA TYPES 56 CONFIGURING COLLECTORS 56 CONFIGURE DISTRIBUTORS 58 CONFIGURE STREAM 59 NOZZLE SETUP 61 OPERATION 61

SAP HANA MEDIATOR 63

OVERVIEW 63 EXTRACTION APPROACH 63 SAP HANA PORT DEFAULTS 63 PRE-REQUISITES 63 MEDIATOR CREATION AND CONFIGURATION 63 CREATE NEW SOURCE 63 CONNECT TO SOURCE 64 SELECT DESTINATION 65 CONNECT TO DESTINATION 65 SELECT DATA TYPES 65 CONFIGURING COLLECTORS 66 CONFIGURE DISTRIBUTORS 67 CONFIGURE STREAM 68

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

vi

OPERATIONS 70

STARTING METRIC EXTRACTION 70 STOPPING METRIC EXTRACTION 70 RUNNING SINGLE EXTRACTIONS 70 EDITING AN EXISTING STREAM 72

PREDICTIVE INSIGHTS CSV OUTPUT 74

FILE DETAILS 74 TIMESTAMPS 76

PREDICTIVE INSIGHTS lsquoMODELrsquo FILES 77

GROUP INFORMATION 77 OBTAINING METRIC NAMES FOR PI MEDIATION TOOL 78

PREDICTIVE INSIGHTS REST OUTPUT 80

LOGS 81 CHANGING LOG LEVELS 81

SUPPORT 83

APPENDIX 1 SAP HANA METRICS 84

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

7

Overview

IBM Operations Analytics Predictive Insights analyzes data and learns the normal behavior of a

system through KPI metric data streams This includes data from various sources including

metrics gathered by industry performance and network monitoring solutions A mediation pack

helps users quickly integrate the KPI data streams by providing a default model to integrate a

particular performance manager

IBM Operations Analytics Predictive Insights Mediation Pack Group 2 known in the rest of this

document as the lsquoMediation Packrsquo provides a pre-built model as a starting point to facilitate data

ingestion Predictive Insights creates a performance model that it uses to detect or forecast

behavior outside the modeled range including generating an alarm when an anomalous behavior

occurs Doing so enables organizations to pro-actively avoid system and solution outages before

users are impacted

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

8

The Mediation Pack provides the capability to extract performance data and metrics from the

following sources

bull SAP Hana DB

bull CA APM (Introscope)

bull MicroFocus NNMi

bull AWS Cloudwatch

bull Pivotal Cloud Foundry

bull Dynatrace Managed

bull AppDynamics

bull Oracle OEM

and deliver that data to IBM Operations Analytics Predictive Insights to perform analysis This

document will cover the installation and setup of Mediation Pack for each of the supported source-

data technologies

The figure shows an overall view of the mediation solution in context

Metrics are extracted by Collectors flow through to CSV Distributors which produce CSV files

The CSV files can then be made available to IBM PI for ingestion in the normal way Alternatively

any Collector can be routed to a PI REST Distributor which will deliver data to PI via its REST

interface

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

9

Prerequisite Information

Item Description

Name Mediation Pack Group 2

Predictive Insights supported

versions

v136 and higher Note IBM PI continues to support the CSV

format for ingestion and so earlier versions of PI ie 133 will

also work)

Predictive Insights Information IBM Operations Analytics (demo trial resources)

httpswwwibmcomus-enmarketplaceit-operations-

analytics

IBM Operations Analytics Community (docs videos

mediation packs) httpsdeveloperibmcomitoa

IBM Operations Analytics Predictive Insights

documentation

httpswwwibmcomsupportknowledgecenterSSJQQ3

Solution Name supported

versions

Min version Max Version

CA APM (Introscope) 9x 107

SAP Hana 10 10

AWS Cloudwatch EC2 REST

API 2016-11-

15

Cloudwatch

111

Same as Min

version

MF NNMi 1040x 1040x

Pivotal Cloud Foundry 233 233

Dynatrace Managed Dec 2018 Jun 2019

AppDynamics 41x 459

Oracle OEM 13 13

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

10

Solution Name Information IBM Operations Analytics Predictive Insights Mediation Pack

Group 2

Filename mediation_pack_group2-27-ibmpi-ltbuildNumbergtzip

Supported Operating Systems Linux

RHEL 6x 7x 3264bit

CentOS 6x7x 3264bit

Java Java18+ is required There is a version included in the release

package Note Mediators do NOT run on IBM JVM (the one

delivered with PI at time of writing)

Web Browsers A modern HTML5CSS3Javascript browser Versions of

Chrome Firefox or Safari since 2012 should work Previous

browser versions may not work well or at all

Note Some versions of Internet Explorer (ie 9 or earlier)

have exhibited issues Please consider a more modern

browser to ensure full functionality of the product

Ports Port 7457 (default but configurable) is required by

Mediators

Outbound ports per Mediator Type

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

11

SWP Mediators Overview

The metric data flow is generally from a collection component (a lsquoCollectorrsquo aka lsquoMediatorrsquo)

specific to the data source to an output component (a lsquoDistributorrsquo) Two distributors types are

support 1 CSV - writes metrics to standard IBM Operations Analytics Predictive Insights CSV files

2 REST - posts metric data (as JSON) to the IBM Operations Analytics Predictive Insights REST interface

Whilst each Mediator requires configuration specific to its source all the data flows through to a

corresponding Predictive Insights distributor The configurations approach for each of the

sources will be described in this document

The Mediators as part of the Mediation pack usually will be installed on the same host as

Predictive Insights This makes it easier to deliver the data from the Mediators to Predictive

Insights If the Mediation pack is installed on a different host then mechanisms must be put in

place to ensure that Predictive Insights can get at that CSV data eg shared disk drivesNFS

FTP-type jobs run on a schedule to retrieve files How to establish such mechanisms is beyond

the scope of this document In the case of REST distribution SWP does not have to be co-

located with Predictive Insights

Key steps to setting up Mediators

bull Installation

bull Configure one or more streams which extract the data from the supported sources

bull Configure the lsquoPredictive Insightsrsquo distributor stage for each stream choosing

between CSV and REST output

bull Start the desired metric extraction lsquotasksrsquo for each of the data sources This

should result in Predictive Insights CSV files being produced or JSON data

being posted to the IBM PI REST interface

bull Note Setting upconfiguring IBM PI REST interface is beyond the

scope of this document For details on this please refer to the

corresponding IBM PI documentation

bull In the case of CSV integration configure Predictive Insights for ingestion

using its lsquoMediation Toolrsquo in the normal way Sample Predictive Insights

lsquomodelrsquo files are provided as a starting point The lsquoMediation Toolrsquo is not

used for PI REST integration

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

12

Installing and Configuring

Zip file contents

The file mediation_pack_group2_v27-ibmpi- ltbuildgtzip contains the following

bull streamweaver-ibmpi-27ibmpiltbuildgt-1noarchrpm contains the main SWPMediators executable and associated collateral

bull docs containing the User Guides

o for lsquostreamweaverrsquo (this document)

o for Spl_AppD_Dyna_Ater_MedPack_v15_Revltrevisiongt

bull license containing license files

bull Spl_AppD_Dyna_Ater_MedPack_v15ltbuildgtrpm release with support for

o Dynatrace

o DCRum

o Aternity

o Splunk

o ExtraHop

o CA UIM

o MF APM

o NetIQ AppManager

o Note this package also contains support for AppDynamics Oracle OEM and CA APMIntroscope - these have been deprecated in 1x replaced by the Mediators in the main streamweaver 27 version

Install procedure

New Installation

The following section provides information for installing and configuring the Mediation Pack on

your IBM Operations Analytics Predictive Insights system

Normal lsquorpmrsquo conventions are followed

1 Verify Hardware amp Software Requirements

2 Verify user account privilege requirements

i The installation should be performed as the lsquorootrsquo user or user which can sudo ltgt

with root privileges This is required to create the Linux service

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

13

ii The installer will install as scadmindb2iadm (to match PI)

3 unzip mediation pack file

unzip mediation_pack_group2_v27-ibmpi-ltbuildgtzip

4 Install the bundled JVM

rpm -i --nodeps opstream-jre-18ibmpiltbuildgt-1x86_64rpm

5 Install SWP Mediators Locate the rpm file in the expanded zip directory

rpm -i streamweaver-ibmpi-27ibmpiltbuildgt-

1noarchrpm

After this you should see an indication of successful installation eg

SWP Mediators are installed by default to

optIBMstreamweaver-ibmpi

6 If you used root account exit and return to lsquoscadminrsquo user

Notes

Installing to an alternative location

You can install to a location other than optIBM as follows

sudo rpm -i --relocate optIBM=ltalternativepathgt streamweaver-

ibmpi-27ibmpiltbuildgt-1noarchrpm

Running as non-`scadmin` user

As part of the install if not already present in the system a `scadmin` account and lsquodb2iadm1` group are created for consistency with the default IBM PI installation and related file-ownerships

If you wish to run as a non-`scadmin` user

Change ownership of all the files from down eg

chown -R differentuserdifferentgroup optIBMstreamweaver-ibmpi

Edit the initd startup script

etcinitdstreamweaver-api

Locate and edit the line specifying the DAEMON_USER

[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=scadmin

Change the default `scadmin` to be your desired user eg

[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=differentuser

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

14

Co-existence of SWP-based and Accelerator-based mediators

If you are using the earlier Accelerator-based version along with this SWP-based version

note that they use the same default port setting of 7457 In this situation you should change the

default for SWP to something else You can do this by locating and editing the installed config

file in

optIBMstreamweaver-ibmpi confswpconf

and locate uncomment and change the line with port = 7457 to some other port

Settings for the Core Services services

Settings for the UI Service

ui

How the UI Services binds to a socket

connection

Hostname or IP Address to listen on The default

0000 causes

the REST API to listen on all interfaces

host = 0000

The port number to listen on

port = 7457

to some other free port

Also By default the UI listens on 7457 and the REST API listens on 7456 The UI assumes that the REST port is 1 less than the UI port Having made the above config change the following config entries are required and must be explicitly added to swpconf at the end of the file

httpport = ltrest-port-numbergt must be one less than UI port this defines the

port that REST API listens on lagomservicesswpallinone = httplocalhostltrest-port-

number server-side clients of REST service use this to locate it NOTE quotes are important here

Data directories

If you wish to relocate the various working directories (eg logs data) edit the swpconf file and modify the relevant entries in the ldquopathsrsquo section

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

15

Installation users

SWPMediators is installed by root as usergroup scadmindb2iadm1 to align the default PI usergroups

Starting SWP system

SWP contains Kafka Cassandra and the core They are all run under the lsquoswprsquo service

bull As lsquorootrsquo execute

service streamweaver-ibmpi start

In the usual way running lsquostatusrsquo on the service should give an indication that the main

SWP demon is operational

service streamweaver-ibmpi status

Once the SWP demon is running you use the SWP command line utility to monitor the

status of the child processes

Use lsquoswprsquo to observe the status of the system

bull optIBMstreamweaver-ibmpibinswp status

You should see a response similar to the following

In normal operation the five processes should be in Running state

You can also run the status command in a loop (like Linux `watch`) eg

bull optIBMstreamweaver-ibmpibinswp status -w 3

Stopping SWP system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

16

service streamweaver-ibmpi stop

Upgrade installation

service streamweaver-ibmpi stop

rpm -e streamweaver-ibmpi

Install new version as described above

Deployed directory contents

The main directories are created as part of the installation

optIBMstreamweaver-ibmpi

jre

The bundled JRE

bin

lsquoswpdrsquo all-in-one executable and lsquoswprsquo command line

content-packs

lsquoPI mediation lsquopa modelrsquo files and Pivotal CF lsquonozzlersquo components

lib

Library files (jars)

logs

Log files useful for monitoring and debugging system behavior

conf

Configuration files to control SWP

services

Cassandra Kafka and swpallinone (core SWP services)

Configuration files to control streamweaver

dataswp

Extracted metric data is stored under here

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

17

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

18

Accessing the Mediation system

Once installed you can interact with the Mediation system via web-browser By default the

Mediation system listens on port 7457 so for example point your web-browser to

httpltservergt7457

Weaver Pressing lsquoWEAVERrsquo brings you back to the above screen You select the lsquo+rsquo and a list

of source options will appear Any 3rd party tool listed can be selected to begin the configuration process In a similar fashion Destinations can also be configured by selecting the plus icon in the Destinations section

Dashboard Pressing lsquoDASHBOARDrsquo while cause latency measurements (the duration of last collection for each destination mediator) for each stream to be presented

Configuration Pressing lsquoDASHBOARDrsquo shows configuration information for the platform

including particularly log levels

System Logs Pressing lsquoSYSTEM LOGSrsquo presents the latest log entries (the full set of logs and be viewed in the relevant log directory which is described later in this document)

In the subsequent Mediator-specific sections we will give specific examples of

Mediator-gtDistributor Each example will specify a specific kind of distributor ie CSV or REST

You are free to use either distributor in each case

The initial step in each configuration case is to press lsquo+rsquo on the SOURCES and then you should

be presented with a list of available Mediator source types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

19

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

20

AppDynamics Mediator

Overview

Extraction Approach

Metric data is extracted using the AppDynamics REST API

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

AppD Port Defaults

REST 443

Mediator creation and configuration

To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your AppDynamics source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

21

Select Destination

Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single

file stream emerges from each Destination and we donrsquot generally want to mix data from

different Collector instances

Give this destination a unique name This name will also be used as a prefix for the

AppDynamics CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

22

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from AppDynamics is based upon selecting Applications Tiers and metric

paths SWP will extract all metrics for the supplied paths

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

23

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)

when data for the next time interval arrives

You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open

for the specified number of periods (as defined by the Maxium Time Window Per File) This is

useful when you want to wait for late data The trade-off is that the files are only made available

to PI after that lag period - thereby increasing end to end latency

For example if you had 5min files and set the max lag to 2 yoursquod have current file open and

two older files leading to up to 15 min latency

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

24

AWS Cloudwatch Mediator

Overview

This mediator extracts Metric data from AWS Cloudwatch system

Extraction Approach

Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference

Pre-requisites

You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users

The Endpoint regions are shown in the following table For the subsequent SWP configuration

you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account

has access to metric data

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

25

T

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

26

Mediator creation and configuration

To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the Source column to add a new origin

Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for AWS Cloudwatch account

Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least

one Region Multiple regions can be entered by separating each with a comma

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the AWS

CloudWatch environment)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

27

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

28

Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics

in those categories SWP will extract all metrics for those selected metric tables For IBM PI

only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are

selected

The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide

convenient resource groupings If you want to scope your metric extraction to particular tagged

instances specify those tags here as comma separated tagname=tagvalue pairs

For more details on AWS tagging see the AWS documentation

The optimum settings for the other configuration values depend on your AWS accountdata

availability etc The defaults should be sufficient to get you operational

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

29

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

30

CA APM (Introscope) Mediator

Overview

CA APM was formerly known as Wily Introscope

Metric extraction from CA APM is based upon configuring the system to extract data for specific

Hosts Metric data will be collected for all Agents and Processes on each selected host

Extraction Approach

Metric data is extracted using a JDBC interface

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

CA APM Port Defaults

JDBC 5001

Note

The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used

Mediator creation and configuration

To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

and then press lsquo+rsquo on the Sources column to add a new sources

Provide a name for the new source instance

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

31

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your CA APM DB

The Metadata Window Width and Latency control time window of metric data which is

extracted to determine the available monitored hosts for selection

Note It is also possible to configure selection to use regexp expressions which are used directly

in the JDBC queries sent to CA APM Depending on whether or not you check the regexp

button you will be presented with a list of Agents(Hosts) to select from or be required to

provide regexps for Agents and Metric selection - see next section

The TimeZone is that of the CA APM server This must be configured to ensure the metric data

is correctly mapped to UTC

Enter the connection information for the CA APM Host

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the CA

APM host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

32

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name

Give this destination a unique name This name will also be used as a prefix for the caapm CSV

files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

33

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from CA APM is based upon selecting the hosts and SWP will extract all

metrics for that host

Note If you had chosen to configure via regexp the list of hosts panel will be empty

You must provide a set of comma separated regexp pairs - one pair per line in the RegExp

panel eg

User

These are regexps for CA APM Agents and Metrics and are used directly in the queries eg

select from metric_data where agent= and metric=User

See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

34

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Note Sometimes depending on the responsiveness of the CA-APM system it may take a few

seconds for the list of hosts to appear Clicking in the panel often helps with refresh

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

35

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

36

Dynatrace Managed

Overview

This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api

Extraction Approach

Pre-requisites

The hostport and REST api connection token for your Dynatrace Managed system (see your

Admin andor Dynatrace API - Authentication)

Mediator creation and configuration

To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new origin

Press lsquo+rsquo on the SOURCES column to add a new source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

37

Now provide the connectivity information for your Dynatrace Managed system

Enter your Dynatrace Managed credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

Dynatrace Managed environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to produce IBM PI CSV files

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

38

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Select the desired Dynatrace Managed metric sources

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

39

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 3: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

iii

CONTENTS

Table of Contents

MEDIATION PACK GROUP 2 I

CONTENTS III

OVERVIEW 7

PREREQUISITE INFORMATION 9

SWP MEDIATORS OVERVIEW 11

KEY STEPS TO SETTING UP MEDIATORS 11

INSTALLING AND CONFIGURING 12

ZIP FILE CONTENTS 12 INSTALL PROCEDURE 12 NEW INSTALLATION 12 NOTES 13 STARTING SWP SYSTEM 15 STOPPING SWP SYSTEM 15 UPGRADE INSTALLATION 16 DEPLOYED DIRECTORY CONTENTS 16 ACCESSING THE MEDIATION SYSTEM 18

APPDYNAMICS MEDIATOR 20

OVERVIEW 20 MEDIATOR CREATION AND CONFIGURATION 20 SELECT SOURCE 20 CONNECT TO SOURCE 20 SELECT DESTINATION 21 CONNECT TO DESTINATION 21 SELECT DATA TYPES 21 CONFIGURING COLLECTORS 22 CONFIGURING DISTRIBUTORS 23 SAVE STREAM 23

AWS CLOUDWATCH MEDIATOR 24

OVERVIEW 24 EXTRACTION APPROACH 24 PRE-REQUISITES 24 MEDIATOR CREATION AND CONFIGURATION 26 CREATE NEW SOURCE 26

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

iv

CONNECT TO SOURCE 26 SELECT DESTINATION 27 CONNECT TO DESTINATION 27 SELECT DATA TYPES 27 CONFIGURING COLLECTORS 27 CONFIGURE DISTRIBUTORS 29 CONFIGURE STREAM 29

CA APM (INTROSCOPE) MEDIATOR 30

OVERVIEW 30 NOTE 30 MEDIATOR CREATION AND CONFIGURATION 30 SELECT SOURCE 30 CONNECT TO SOURCE 31 SELECT DESTINATION 32 CONNECT TO DESTINATION 32 SELECT DATA TYPES 32 CONFIGURING COLLECTORS 33 CONFIGURING DISTRIBUTORS 35 CONFIGURE STREAM 35

DYNATRACE MANAGED 36

OVERVIEW 36 EXTRACTION APPROACH 36 PRE-REQUISITES 36 MEDIATOR CREATION AND CONFIGURATION 36 CREATE NEW ORIGIN 36 CONNECT TO SOURCE 36 SELECT DESTINATION 37 CONNECT TO DESTINATION 37 SELECT DATA TYPES 38 CONFIGURING COLLECTORS 38 CONFIGURE DISTRIBUTORS 39 CONFIGURE STREAM 40

MF NNMI MEDIATOR 41

OVERVIEW 41 EXTRACTION APPROACH 41 PRE-REQUISITES 41 MEDIATOR CREATION AND CONFIGURATION 43 CREATE NEW SOURCE 43 CONNECT TO SOURCE 43 SELECT DESTINATION 44 CONNECT TO DESTINATION 44 SELECT DATA TYPES 44 CONFIGURING COLLECTORS 45

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

v

CONFIGURE DISTRIBUTORS 46 CONFIGURE STREAM 47

ORACLE OEM MEDIATOR 48

OVERVIEW 48 MEDIATOR CREATION AND CONFIGURATION 48 SELECT SOURCE 48 CONNECT TO SOURCE 48 SELECT DESTINATION 49 CONNECT TO DESTINATION 49 SELECT DATA TYPES 50 CONFIGURING COLLECTORS 50 CONFIGURING DISTRIBUTORS 51 SAVE STREAM 51

PIVOTAL CLOUD FOUNDRY MEDIATOR 52

OVERVIEW 52 EXTRACTION APPROACH 52 PRE-REQUISITES 52 OVERALL SETUP 53 MEDIATOR CREATION AND CONFIGURATION 54 CREATE NEW SOURCE 54 CONNECT TO SOURCE 54 SELECT DESTINATION 55 CONNECT TO DESTINATION 56 SELECT DATA TYPES 56 CONFIGURING COLLECTORS 56 CONFIGURE DISTRIBUTORS 58 CONFIGURE STREAM 59 NOZZLE SETUP 61 OPERATION 61

SAP HANA MEDIATOR 63

OVERVIEW 63 EXTRACTION APPROACH 63 SAP HANA PORT DEFAULTS 63 PRE-REQUISITES 63 MEDIATOR CREATION AND CONFIGURATION 63 CREATE NEW SOURCE 63 CONNECT TO SOURCE 64 SELECT DESTINATION 65 CONNECT TO DESTINATION 65 SELECT DATA TYPES 65 CONFIGURING COLLECTORS 66 CONFIGURE DISTRIBUTORS 67 CONFIGURE STREAM 68

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

vi

OPERATIONS 70

STARTING METRIC EXTRACTION 70 STOPPING METRIC EXTRACTION 70 RUNNING SINGLE EXTRACTIONS 70 EDITING AN EXISTING STREAM 72

PREDICTIVE INSIGHTS CSV OUTPUT 74

FILE DETAILS 74 TIMESTAMPS 76

PREDICTIVE INSIGHTS lsquoMODELrsquo FILES 77

GROUP INFORMATION 77 OBTAINING METRIC NAMES FOR PI MEDIATION TOOL 78

PREDICTIVE INSIGHTS REST OUTPUT 80

LOGS 81 CHANGING LOG LEVELS 81

SUPPORT 83

APPENDIX 1 SAP HANA METRICS 84

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

7

Overview

IBM Operations Analytics Predictive Insights analyzes data and learns the normal behavior of a

system through KPI metric data streams This includes data from various sources including

metrics gathered by industry performance and network monitoring solutions A mediation pack

helps users quickly integrate the KPI data streams by providing a default model to integrate a

particular performance manager

IBM Operations Analytics Predictive Insights Mediation Pack Group 2 known in the rest of this

document as the lsquoMediation Packrsquo provides a pre-built model as a starting point to facilitate data

ingestion Predictive Insights creates a performance model that it uses to detect or forecast

behavior outside the modeled range including generating an alarm when an anomalous behavior

occurs Doing so enables organizations to pro-actively avoid system and solution outages before

users are impacted

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

8

The Mediation Pack provides the capability to extract performance data and metrics from the

following sources

bull SAP Hana DB

bull CA APM (Introscope)

bull MicroFocus NNMi

bull AWS Cloudwatch

bull Pivotal Cloud Foundry

bull Dynatrace Managed

bull AppDynamics

bull Oracle OEM

and deliver that data to IBM Operations Analytics Predictive Insights to perform analysis This

document will cover the installation and setup of Mediation Pack for each of the supported source-

data technologies

The figure shows an overall view of the mediation solution in context

Metrics are extracted by Collectors flow through to CSV Distributors which produce CSV files

The CSV files can then be made available to IBM PI for ingestion in the normal way Alternatively

any Collector can be routed to a PI REST Distributor which will deliver data to PI via its REST

interface

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

9

Prerequisite Information

Item Description

Name Mediation Pack Group 2

Predictive Insights supported

versions

v136 and higher Note IBM PI continues to support the CSV

format for ingestion and so earlier versions of PI ie 133 will

also work)

Predictive Insights Information IBM Operations Analytics (demo trial resources)

httpswwwibmcomus-enmarketplaceit-operations-

analytics

IBM Operations Analytics Community (docs videos

mediation packs) httpsdeveloperibmcomitoa

IBM Operations Analytics Predictive Insights

documentation

httpswwwibmcomsupportknowledgecenterSSJQQ3

Solution Name supported

versions

Min version Max Version

CA APM (Introscope) 9x 107

SAP Hana 10 10

AWS Cloudwatch EC2 REST

API 2016-11-

15

Cloudwatch

111

Same as Min

version

MF NNMi 1040x 1040x

Pivotal Cloud Foundry 233 233

Dynatrace Managed Dec 2018 Jun 2019

AppDynamics 41x 459

Oracle OEM 13 13

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

10

Solution Name Information IBM Operations Analytics Predictive Insights Mediation Pack

Group 2

Filename mediation_pack_group2-27-ibmpi-ltbuildNumbergtzip

Supported Operating Systems Linux

RHEL 6x 7x 3264bit

CentOS 6x7x 3264bit

Java Java18+ is required There is a version included in the release

package Note Mediators do NOT run on IBM JVM (the one

delivered with PI at time of writing)

Web Browsers A modern HTML5CSS3Javascript browser Versions of

Chrome Firefox or Safari since 2012 should work Previous

browser versions may not work well or at all

Note Some versions of Internet Explorer (ie 9 or earlier)

have exhibited issues Please consider a more modern

browser to ensure full functionality of the product

Ports Port 7457 (default but configurable) is required by

Mediators

Outbound ports per Mediator Type

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

11

SWP Mediators Overview

The metric data flow is generally from a collection component (a lsquoCollectorrsquo aka lsquoMediatorrsquo)

specific to the data source to an output component (a lsquoDistributorrsquo) Two distributors types are

support 1 CSV - writes metrics to standard IBM Operations Analytics Predictive Insights CSV files

2 REST - posts metric data (as JSON) to the IBM Operations Analytics Predictive Insights REST interface

Whilst each Mediator requires configuration specific to its source all the data flows through to a

corresponding Predictive Insights distributor The configurations approach for each of the

sources will be described in this document

The Mediators as part of the Mediation pack usually will be installed on the same host as

Predictive Insights This makes it easier to deliver the data from the Mediators to Predictive

Insights If the Mediation pack is installed on a different host then mechanisms must be put in

place to ensure that Predictive Insights can get at that CSV data eg shared disk drivesNFS

FTP-type jobs run on a schedule to retrieve files How to establish such mechanisms is beyond

the scope of this document In the case of REST distribution SWP does not have to be co-

located with Predictive Insights

Key steps to setting up Mediators

bull Installation

bull Configure one or more streams which extract the data from the supported sources

bull Configure the lsquoPredictive Insightsrsquo distributor stage for each stream choosing

between CSV and REST output

bull Start the desired metric extraction lsquotasksrsquo for each of the data sources This

should result in Predictive Insights CSV files being produced or JSON data

being posted to the IBM PI REST interface

bull Note Setting upconfiguring IBM PI REST interface is beyond the

scope of this document For details on this please refer to the

corresponding IBM PI documentation

bull In the case of CSV integration configure Predictive Insights for ingestion

using its lsquoMediation Toolrsquo in the normal way Sample Predictive Insights

lsquomodelrsquo files are provided as a starting point The lsquoMediation Toolrsquo is not

used for PI REST integration

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

12

Installing and Configuring

Zip file contents

The file mediation_pack_group2_v27-ibmpi- ltbuildgtzip contains the following

bull streamweaver-ibmpi-27ibmpiltbuildgt-1noarchrpm contains the main SWPMediators executable and associated collateral

bull docs containing the User Guides

o for lsquostreamweaverrsquo (this document)

o for Spl_AppD_Dyna_Ater_MedPack_v15_Revltrevisiongt

bull license containing license files

bull Spl_AppD_Dyna_Ater_MedPack_v15ltbuildgtrpm release with support for

o Dynatrace

o DCRum

o Aternity

o Splunk

o ExtraHop

o CA UIM

o MF APM

o NetIQ AppManager

o Note this package also contains support for AppDynamics Oracle OEM and CA APMIntroscope - these have been deprecated in 1x replaced by the Mediators in the main streamweaver 27 version

Install procedure

New Installation

The following section provides information for installing and configuring the Mediation Pack on

your IBM Operations Analytics Predictive Insights system

Normal lsquorpmrsquo conventions are followed

1 Verify Hardware amp Software Requirements

2 Verify user account privilege requirements

i The installation should be performed as the lsquorootrsquo user or user which can sudo ltgt

with root privileges This is required to create the Linux service

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

13

ii The installer will install as scadmindb2iadm (to match PI)

3 unzip mediation pack file

unzip mediation_pack_group2_v27-ibmpi-ltbuildgtzip

4 Install the bundled JVM

rpm -i --nodeps opstream-jre-18ibmpiltbuildgt-1x86_64rpm

5 Install SWP Mediators Locate the rpm file in the expanded zip directory

rpm -i streamweaver-ibmpi-27ibmpiltbuildgt-

1noarchrpm

After this you should see an indication of successful installation eg

SWP Mediators are installed by default to

optIBMstreamweaver-ibmpi

6 If you used root account exit and return to lsquoscadminrsquo user

Notes

Installing to an alternative location

You can install to a location other than optIBM as follows

sudo rpm -i --relocate optIBM=ltalternativepathgt streamweaver-

ibmpi-27ibmpiltbuildgt-1noarchrpm

Running as non-`scadmin` user

As part of the install if not already present in the system a `scadmin` account and lsquodb2iadm1` group are created for consistency with the default IBM PI installation and related file-ownerships

If you wish to run as a non-`scadmin` user

Change ownership of all the files from down eg

chown -R differentuserdifferentgroup optIBMstreamweaver-ibmpi

Edit the initd startup script

etcinitdstreamweaver-api

Locate and edit the line specifying the DAEMON_USER

[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=scadmin

Change the default `scadmin` to be your desired user eg

[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=differentuser

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

14

Co-existence of SWP-based and Accelerator-based mediators

If you are using the earlier Accelerator-based version along with this SWP-based version

note that they use the same default port setting of 7457 In this situation you should change the

default for SWP to something else You can do this by locating and editing the installed config

file in

optIBMstreamweaver-ibmpi confswpconf

and locate uncomment and change the line with port = 7457 to some other port

Settings for the Core Services services

Settings for the UI Service

ui

How the UI Services binds to a socket

connection

Hostname or IP Address to listen on The default

0000 causes

the REST API to listen on all interfaces

host = 0000

The port number to listen on

port = 7457

to some other free port

Also By default the UI listens on 7457 and the REST API listens on 7456 The UI assumes that the REST port is 1 less than the UI port Having made the above config change the following config entries are required and must be explicitly added to swpconf at the end of the file

httpport = ltrest-port-numbergt must be one less than UI port this defines the

port that REST API listens on lagomservicesswpallinone = httplocalhostltrest-port-

number server-side clients of REST service use this to locate it NOTE quotes are important here

Data directories

If you wish to relocate the various working directories (eg logs data) edit the swpconf file and modify the relevant entries in the ldquopathsrsquo section

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

15

Installation users

SWPMediators is installed by root as usergroup scadmindb2iadm1 to align the default PI usergroups

Starting SWP system

SWP contains Kafka Cassandra and the core They are all run under the lsquoswprsquo service

bull As lsquorootrsquo execute

service streamweaver-ibmpi start

In the usual way running lsquostatusrsquo on the service should give an indication that the main

SWP demon is operational

service streamweaver-ibmpi status

Once the SWP demon is running you use the SWP command line utility to monitor the

status of the child processes

Use lsquoswprsquo to observe the status of the system

bull optIBMstreamweaver-ibmpibinswp status

You should see a response similar to the following

In normal operation the five processes should be in Running state

You can also run the status command in a loop (like Linux `watch`) eg

bull optIBMstreamweaver-ibmpibinswp status -w 3

Stopping SWP system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

16

service streamweaver-ibmpi stop

Upgrade installation

service streamweaver-ibmpi stop

rpm -e streamweaver-ibmpi

Install new version as described above

Deployed directory contents

The main directories are created as part of the installation

optIBMstreamweaver-ibmpi

jre

The bundled JRE

bin

lsquoswpdrsquo all-in-one executable and lsquoswprsquo command line

content-packs

lsquoPI mediation lsquopa modelrsquo files and Pivotal CF lsquonozzlersquo components

lib

Library files (jars)

logs

Log files useful for monitoring and debugging system behavior

conf

Configuration files to control SWP

services

Cassandra Kafka and swpallinone (core SWP services)

Configuration files to control streamweaver

dataswp

Extracted metric data is stored under here

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

17

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

18

Accessing the Mediation system

Once installed you can interact with the Mediation system via web-browser By default the

Mediation system listens on port 7457 so for example point your web-browser to

httpltservergt7457

Weaver Pressing lsquoWEAVERrsquo brings you back to the above screen You select the lsquo+rsquo and a list

of source options will appear Any 3rd party tool listed can be selected to begin the configuration process In a similar fashion Destinations can also be configured by selecting the plus icon in the Destinations section

Dashboard Pressing lsquoDASHBOARDrsquo while cause latency measurements (the duration of last collection for each destination mediator) for each stream to be presented

Configuration Pressing lsquoDASHBOARDrsquo shows configuration information for the platform

including particularly log levels

System Logs Pressing lsquoSYSTEM LOGSrsquo presents the latest log entries (the full set of logs and be viewed in the relevant log directory which is described later in this document)

In the subsequent Mediator-specific sections we will give specific examples of

Mediator-gtDistributor Each example will specify a specific kind of distributor ie CSV or REST

You are free to use either distributor in each case

The initial step in each configuration case is to press lsquo+rsquo on the SOURCES and then you should

be presented with a list of available Mediator source types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

19

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

20

AppDynamics Mediator

Overview

Extraction Approach

Metric data is extracted using the AppDynamics REST API

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

AppD Port Defaults

REST 443

Mediator creation and configuration

To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your AppDynamics source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

21

Select Destination

Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single

file stream emerges from each Destination and we donrsquot generally want to mix data from

different Collector instances

Give this destination a unique name This name will also be used as a prefix for the

AppDynamics CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

22

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from AppDynamics is based upon selecting Applications Tiers and metric

paths SWP will extract all metrics for the supplied paths

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

23

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)

when data for the next time interval arrives

You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open

for the specified number of periods (as defined by the Maxium Time Window Per File) This is

useful when you want to wait for late data The trade-off is that the files are only made available

to PI after that lag period - thereby increasing end to end latency

For example if you had 5min files and set the max lag to 2 yoursquod have current file open and

two older files leading to up to 15 min latency

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

24

AWS Cloudwatch Mediator

Overview

This mediator extracts Metric data from AWS Cloudwatch system

Extraction Approach

Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference

Pre-requisites

You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users

The Endpoint regions are shown in the following table For the subsequent SWP configuration

you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account

has access to metric data

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

25

T

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

26

Mediator creation and configuration

To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the Source column to add a new origin

Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for AWS Cloudwatch account

Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least

one Region Multiple regions can be entered by separating each with a comma

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the AWS

CloudWatch environment)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

27

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

28

Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics

in those categories SWP will extract all metrics for those selected metric tables For IBM PI

only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are

selected

The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide

convenient resource groupings If you want to scope your metric extraction to particular tagged

instances specify those tags here as comma separated tagname=tagvalue pairs

For more details on AWS tagging see the AWS documentation

The optimum settings for the other configuration values depend on your AWS accountdata

availability etc The defaults should be sufficient to get you operational

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

29

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

30

CA APM (Introscope) Mediator

Overview

CA APM was formerly known as Wily Introscope

Metric extraction from CA APM is based upon configuring the system to extract data for specific

Hosts Metric data will be collected for all Agents and Processes on each selected host

Extraction Approach

Metric data is extracted using a JDBC interface

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

CA APM Port Defaults

JDBC 5001

Note

The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used

Mediator creation and configuration

To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

and then press lsquo+rsquo on the Sources column to add a new sources

Provide a name for the new source instance

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

31

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your CA APM DB

The Metadata Window Width and Latency control time window of metric data which is

extracted to determine the available monitored hosts for selection

Note It is also possible to configure selection to use regexp expressions which are used directly

in the JDBC queries sent to CA APM Depending on whether or not you check the regexp

button you will be presented with a list of Agents(Hosts) to select from or be required to

provide regexps for Agents and Metric selection - see next section

The TimeZone is that of the CA APM server This must be configured to ensure the metric data

is correctly mapped to UTC

Enter the connection information for the CA APM Host

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the CA

APM host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

32

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name

Give this destination a unique name This name will also be used as a prefix for the caapm CSV

files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

33

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from CA APM is based upon selecting the hosts and SWP will extract all

metrics for that host

Note If you had chosen to configure via regexp the list of hosts panel will be empty

You must provide a set of comma separated regexp pairs - one pair per line in the RegExp

panel eg

User

These are regexps for CA APM Agents and Metrics and are used directly in the queries eg

select from metric_data where agent= and metric=User

See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

34

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Note Sometimes depending on the responsiveness of the CA-APM system it may take a few

seconds for the list of hosts to appear Clicking in the panel often helps with refresh

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

35

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

36

Dynatrace Managed

Overview

This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api

Extraction Approach

Pre-requisites

The hostport and REST api connection token for your Dynatrace Managed system (see your

Admin andor Dynatrace API - Authentication)

Mediator creation and configuration

To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new origin

Press lsquo+rsquo on the SOURCES column to add a new source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

37

Now provide the connectivity information for your Dynatrace Managed system

Enter your Dynatrace Managed credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

Dynatrace Managed environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to produce IBM PI CSV files

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

38

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Select the desired Dynatrace Managed metric sources

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

39

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 4: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

iv

CONNECT TO SOURCE 26 SELECT DESTINATION 27 CONNECT TO DESTINATION 27 SELECT DATA TYPES 27 CONFIGURING COLLECTORS 27 CONFIGURE DISTRIBUTORS 29 CONFIGURE STREAM 29

CA APM (INTROSCOPE) MEDIATOR 30

OVERVIEW 30 NOTE 30 MEDIATOR CREATION AND CONFIGURATION 30 SELECT SOURCE 30 CONNECT TO SOURCE 31 SELECT DESTINATION 32 CONNECT TO DESTINATION 32 SELECT DATA TYPES 32 CONFIGURING COLLECTORS 33 CONFIGURING DISTRIBUTORS 35 CONFIGURE STREAM 35

DYNATRACE MANAGED 36

OVERVIEW 36 EXTRACTION APPROACH 36 PRE-REQUISITES 36 MEDIATOR CREATION AND CONFIGURATION 36 CREATE NEW ORIGIN 36 CONNECT TO SOURCE 36 SELECT DESTINATION 37 CONNECT TO DESTINATION 37 SELECT DATA TYPES 38 CONFIGURING COLLECTORS 38 CONFIGURE DISTRIBUTORS 39 CONFIGURE STREAM 40

MF NNMI MEDIATOR 41

OVERVIEW 41 EXTRACTION APPROACH 41 PRE-REQUISITES 41 MEDIATOR CREATION AND CONFIGURATION 43 CREATE NEW SOURCE 43 CONNECT TO SOURCE 43 SELECT DESTINATION 44 CONNECT TO DESTINATION 44 SELECT DATA TYPES 44 CONFIGURING COLLECTORS 45

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

v

CONFIGURE DISTRIBUTORS 46 CONFIGURE STREAM 47

ORACLE OEM MEDIATOR 48

OVERVIEW 48 MEDIATOR CREATION AND CONFIGURATION 48 SELECT SOURCE 48 CONNECT TO SOURCE 48 SELECT DESTINATION 49 CONNECT TO DESTINATION 49 SELECT DATA TYPES 50 CONFIGURING COLLECTORS 50 CONFIGURING DISTRIBUTORS 51 SAVE STREAM 51

PIVOTAL CLOUD FOUNDRY MEDIATOR 52

OVERVIEW 52 EXTRACTION APPROACH 52 PRE-REQUISITES 52 OVERALL SETUP 53 MEDIATOR CREATION AND CONFIGURATION 54 CREATE NEW SOURCE 54 CONNECT TO SOURCE 54 SELECT DESTINATION 55 CONNECT TO DESTINATION 56 SELECT DATA TYPES 56 CONFIGURING COLLECTORS 56 CONFIGURE DISTRIBUTORS 58 CONFIGURE STREAM 59 NOZZLE SETUP 61 OPERATION 61

SAP HANA MEDIATOR 63

OVERVIEW 63 EXTRACTION APPROACH 63 SAP HANA PORT DEFAULTS 63 PRE-REQUISITES 63 MEDIATOR CREATION AND CONFIGURATION 63 CREATE NEW SOURCE 63 CONNECT TO SOURCE 64 SELECT DESTINATION 65 CONNECT TO DESTINATION 65 SELECT DATA TYPES 65 CONFIGURING COLLECTORS 66 CONFIGURE DISTRIBUTORS 67 CONFIGURE STREAM 68

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

vi

OPERATIONS 70

STARTING METRIC EXTRACTION 70 STOPPING METRIC EXTRACTION 70 RUNNING SINGLE EXTRACTIONS 70 EDITING AN EXISTING STREAM 72

PREDICTIVE INSIGHTS CSV OUTPUT 74

FILE DETAILS 74 TIMESTAMPS 76

PREDICTIVE INSIGHTS lsquoMODELrsquo FILES 77

GROUP INFORMATION 77 OBTAINING METRIC NAMES FOR PI MEDIATION TOOL 78

PREDICTIVE INSIGHTS REST OUTPUT 80

LOGS 81 CHANGING LOG LEVELS 81

SUPPORT 83

APPENDIX 1 SAP HANA METRICS 84

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

7

Overview

IBM Operations Analytics Predictive Insights analyzes data and learns the normal behavior of a

system through KPI metric data streams This includes data from various sources including

metrics gathered by industry performance and network monitoring solutions A mediation pack

helps users quickly integrate the KPI data streams by providing a default model to integrate a

particular performance manager

IBM Operations Analytics Predictive Insights Mediation Pack Group 2 known in the rest of this

document as the lsquoMediation Packrsquo provides a pre-built model as a starting point to facilitate data

ingestion Predictive Insights creates a performance model that it uses to detect or forecast

behavior outside the modeled range including generating an alarm when an anomalous behavior

occurs Doing so enables organizations to pro-actively avoid system and solution outages before

users are impacted

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

8

The Mediation Pack provides the capability to extract performance data and metrics from the

following sources

bull SAP Hana DB

bull CA APM (Introscope)

bull MicroFocus NNMi

bull AWS Cloudwatch

bull Pivotal Cloud Foundry

bull Dynatrace Managed

bull AppDynamics

bull Oracle OEM

and deliver that data to IBM Operations Analytics Predictive Insights to perform analysis This

document will cover the installation and setup of Mediation Pack for each of the supported source-

data technologies

The figure shows an overall view of the mediation solution in context

Metrics are extracted by Collectors flow through to CSV Distributors which produce CSV files

The CSV files can then be made available to IBM PI for ingestion in the normal way Alternatively

any Collector can be routed to a PI REST Distributor which will deliver data to PI via its REST

interface

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

9

Prerequisite Information

Item Description

Name Mediation Pack Group 2

Predictive Insights supported

versions

v136 and higher Note IBM PI continues to support the CSV

format for ingestion and so earlier versions of PI ie 133 will

also work)

Predictive Insights Information IBM Operations Analytics (demo trial resources)

httpswwwibmcomus-enmarketplaceit-operations-

analytics

IBM Operations Analytics Community (docs videos

mediation packs) httpsdeveloperibmcomitoa

IBM Operations Analytics Predictive Insights

documentation

httpswwwibmcomsupportknowledgecenterSSJQQ3

Solution Name supported

versions

Min version Max Version

CA APM (Introscope) 9x 107

SAP Hana 10 10

AWS Cloudwatch EC2 REST

API 2016-11-

15

Cloudwatch

111

Same as Min

version

MF NNMi 1040x 1040x

Pivotal Cloud Foundry 233 233

Dynatrace Managed Dec 2018 Jun 2019

AppDynamics 41x 459

Oracle OEM 13 13

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

10

Solution Name Information IBM Operations Analytics Predictive Insights Mediation Pack

Group 2

Filename mediation_pack_group2-27-ibmpi-ltbuildNumbergtzip

Supported Operating Systems Linux

RHEL 6x 7x 3264bit

CentOS 6x7x 3264bit

Java Java18+ is required There is a version included in the release

package Note Mediators do NOT run on IBM JVM (the one

delivered with PI at time of writing)

Web Browsers A modern HTML5CSS3Javascript browser Versions of

Chrome Firefox or Safari since 2012 should work Previous

browser versions may not work well or at all

Note Some versions of Internet Explorer (ie 9 or earlier)

have exhibited issues Please consider a more modern

browser to ensure full functionality of the product

Ports Port 7457 (default but configurable) is required by

Mediators

Outbound ports per Mediator Type

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

11

SWP Mediators Overview

The metric data flow is generally from a collection component (a lsquoCollectorrsquo aka lsquoMediatorrsquo)

specific to the data source to an output component (a lsquoDistributorrsquo) Two distributors types are

support 1 CSV - writes metrics to standard IBM Operations Analytics Predictive Insights CSV files

2 REST - posts metric data (as JSON) to the IBM Operations Analytics Predictive Insights REST interface

Whilst each Mediator requires configuration specific to its source all the data flows through to a

corresponding Predictive Insights distributor The configurations approach for each of the

sources will be described in this document

The Mediators as part of the Mediation pack usually will be installed on the same host as

Predictive Insights This makes it easier to deliver the data from the Mediators to Predictive

Insights If the Mediation pack is installed on a different host then mechanisms must be put in

place to ensure that Predictive Insights can get at that CSV data eg shared disk drivesNFS

FTP-type jobs run on a schedule to retrieve files How to establish such mechanisms is beyond

the scope of this document In the case of REST distribution SWP does not have to be co-

located with Predictive Insights

Key steps to setting up Mediators

bull Installation

bull Configure one or more streams which extract the data from the supported sources

bull Configure the lsquoPredictive Insightsrsquo distributor stage for each stream choosing

between CSV and REST output

bull Start the desired metric extraction lsquotasksrsquo for each of the data sources This

should result in Predictive Insights CSV files being produced or JSON data

being posted to the IBM PI REST interface

bull Note Setting upconfiguring IBM PI REST interface is beyond the

scope of this document For details on this please refer to the

corresponding IBM PI documentation

bull In the case of CSV integration configure Predictive Insights for ingestion

using its lsquoMediation Toolrsquo in the normal way Sample Predictive Insights

lsquomodelrsquo files are provided as a starting point The lsquoMediation Toolrsquo is not

used for PI REST integration

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

12

Installing and Configuring

Zip file contents

The file mediation_pack_group2_v27-ibmpi- ltbuildgtzip contains the following

bull streamweaver-ibmpi-27ibmpiltbuildgt-1noarchrpm contains the main SWPMediators executable and associated collateral

bull docs containing the User Guides

o for lsquostreamweaverrsquo (this document)

o for Spl_AppD_Dyna_Ater_MedPack_v15_Revltrevisiongt

bull license containing license files

bull Spl_AppD_Dyna_Ater_MedPack_v15ltbuildgtrpm release with support for

o Dynatrace

o DCRum

o Aternity

o Splunk

o ExtraHop

o CA UIM

o MF APM

o NetIQ AppManager

o Note this package also contains support for AppDynamics Oracle OEM and CA APMIntroscope - these have been deprecated in 1x replaced by the Mediators in the main streamweaver 27 version

Install procedure

New Installation

The following section provides information for installing and configuring the Mediation Pack on

your IBM Operations Analytics Predictive Insights system

Normal lsquorpmrsquo conventions are followed

1 Verify Hardware amp Software Requirements

2 Verify user account privilege requirements

i The installation should be performed as the lsquorootrsquo user or user which can sudo ltgt

with root privileges This is required to create the Linux service

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

13

ii The installer will install as scadmindb2iadm (to match PI)

3 unzip mediation pack file

unzip mediation_pack_group2_v27-ibmpi-ltbuildgtzip

4 Install the bundled JVM

rpm -i --nodeps opstream-jre-18ibmpiltbuildgt-1x86_64rpm

5 Install SWP Mediators Locate the rpm file in the expanded zip directory

rpm -i streamweaver-ibmpi-27ibmpiltbuildgt-

1noarchrpm

After this you should see an indication of successful installation eg

SWP Mediators are installed by default to

optIBMstreamweaver-ibmpi

6 If you used root account exit and return to lsquoscadminrsquo user

Notes

Installing to an alternative location

You can install to a location other than optIBM as follows

sudo rpm -i --relocate optIBM=ltalternativepathgt streamweaver-

ibmpi-27ibmpiltbuildgt-1noarchrpm

Running as non-`scadmin` user

As part of the install if not already present in the system a `scadmin` account and lsquodb2iadm1` group are created for consistency with the default IBM PI installation and related file-ownerships

If you wish to run as a non-`scadmin` user

Change ownership of all the files from down eg

chown -R differentuserdifferentgroup optIBMstreamweaver-ibmpi

Edit the initd startup script

etcinitdstreamweaver-api

Locate and edit the line specifying the DAEMON_USER

[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=scadmin

Change the default `scadmin` to be your desired user eg

[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=differentuser

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

14

Co-existence of SWP-based and Accelerator-based mediators

If you are using the earlier Accelerator-based version along with this SWP-based version

note that they use the same default port setting of 7457 In this situation you should change the

default for SWP to something else You can do this by locating and editing the installed config

file in

optIBMstreamweaver-ibmpi confswpconf

and locate uncomment and change the line with port = 7457 to some other port

Settings for the Core Services services

Settings for the UI Service

ui

How the UI Services binds to a socket

connection

Hostname or IP Address to listen on The default

0000 causes

the REST API to listen on all interfaces

host = 0000

The port number to listen on

port = 7457

to some other free port

Also By default the UI listens on 7457 and the REST API listens on 7456 The UI assumes that the REST port is 1 less than the UI port Having made the above config change the following config entries are required and must be explicitly added to swpconf at the end of the file

httpport = ltrest-port-numbergt must be one less than UI port this defines the

port that REST API listens on lagomservicesswpallinone = httplocalhostltrest-port-

number server-side clients of REST service use this to locate it NOTE quotes are important here

Data directories

If you wish to relocate the various working directories (eg logs data) edit the swpconf file and modify the relevant entries in the ldquopathsrsquo section

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

15

Installation users

SWPMediators is installed by root as usergroup scadmindb2iadm1 to align the default PI usergroups

Starting SWP system

SWP contains Kafka Cassandra and the core They are all run under the lsquoswprsquo service

bull As lsquorootrsquo execute

service streamweaver-ibmpi start

In the usual way running lsquostatusrsquo on the service should give an indication that the main

SWP demon is operational

service streamweaver-ibmpi status

Once the SWP demon is running you use the SWP command line utility to monitor the

status of the child processes

Use lsquoswprsquo to observe the status of the system

bull optIBMstreamweaver-ibmpibinswp status

You should see a response similar to the following

In normal operation the five processes should be in Running state

You can also run the status command in a loop (like Linux `watch`) eg

bull optIBMstreamweaver-ibmpibinswp status -w 3

Stopping SWP system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

16

service streamweaver-ibmpi stop

Upgrade installation

service streamweaver-ibmpi stop

rpm -e streamweaver-ibmpi

Install new version as described above

Deployed directory contents

The main directories are created as part of the installation

optIBMstreamweaver-ibmpi

jre

The bundled JRE

bin

lsquoswpdrsquo all-in-one executable and lsquoswprsquo command line

content-packs

lsquoPI mediation lsquopa modelrsquo files and Pivotal CF lsquonozzlersquo components

lib

Library files (jars)

logs

Log files useful for monitoring and debugging system behavior

conf

Configuration files to control SWP

services

Cassandra Kafka and swpallinone (core SWP services)

Configuration files to control streamweaver

dataswp

Extracted metric data is stored under here

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

17

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

18

Accessing the Mediation system

Once installed you can interact with the Mediation system via web-browser By default the

Mediation system listens on port 7457 so for example point your web-browser to

httpltservergt7457

Weaver Pressing lsquoWEAVERrsquo brings you back to the above screen You select the lsquo+rsquo and a list

of source options will appear Any 3rd party tool listed can be selected to begin the configuration process In a similar fashion Destinations can also be configured by selecting the plus icon in the Destinations section

Dashboard Pressing lsquoDASHBOARDrsquo while cause latency measurements (the duration of last collection for each destination mediator) for each stream to be presented

Configuration Pressing lsquoDASHBOARDrsquo shows configuration information for the platform

including particularly log levels

System Logs Pressing lsquoSYSTEM LOGSrsquo presents the latest log entries (the full set of logs and be viewed in the relevant log directory which is described later in this document)

In the subsequent Mediator-specific sections we will give specific examples of

Mediator-gtDistributor Each example will specify a specific kind of distributor ie CSV or REST

You are free to use either distributor in each case

The initial step in each configuration case is to press lsquo+rsquo on the SOURCES and then you should

be presented with a list of available Mediator source types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

19

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

20

AppDynamics Mediator

Overview

Extraction Approach

Metric data is extracted using the AppDynamics REST API

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

AppD Port Defaults

REST 443

Mediator creation and configuration

To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your AppDynamics source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

21

Select Destination

Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single

file stream emerges from each Destination and we donrsquot generally want to mix data from

different Collector instances

Give this destination a unique name This name will also be used as a prefix for the

AppDynamics CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

22

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from AppDynamics is based upon selecting Applications Tiers and metric

paths SWP will extract all metrics for the supplied paths

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

23

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)

when data for the next time interval arrives

You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open

for the specified number of periods (as defined by the Maxium Time Window Per File) This is

useful when you want to wait for late data The trade-off is that the files are only made available

to PI after that lag period - thereby increasing end to end latency

For example if you had 5min files and set the max lag to 2 yoursquod have current file open and

two older files leading to up to 15 min latency

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

24

AWS Cloudwatch Mediator

Overview

This mediator extracts Metric data from AWS Cloudwatch system

Extraction Approach

Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference

Pre-requisites

You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users

The Endpoint regions are shown in the following table For the subsequent SWP configuration

you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account

has access to metric data

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

25

T

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

26

Mediator creation and configuration

To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the Source column to add a new origin

Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for AWS Cloudwatch account

Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least

one Region Multiple regions can be entered by separating each with a comma

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the AWS

CloudWatch environment)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

27

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

28

Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics

in those categories SWP will extract all metrics for those selected metric tables For IBM PI

only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are

selected

The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide

convenient resource groupings If you want to scope your metric extraction to particular tagged

instances specify those tags here as comma separated tagname=tagvalue pairs

For more details on AWS tagging see the AWS documentation

The optimum settings for the other configuration values depend on your AWS accountdata

availability etc The defaults should be sufficient to get you operational

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

29

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

30

CA APM (Introscope) Mediator

Overview

CA APM was formerly known as Wily Introscope

Metric extraction from CA APM is based upon configuring the system to extract data for specific

Hosts Metric data will be collected for all Agents and Processes on each selected host

Extraction Approach

Metric data is extracted using a JDBC interface

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

CA APM Port Defaults

JDBC 5001

Note

The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used

Mediator creation and configuration

To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

and then press lsquo+rsquo on the Sources column to add a new sources

Provide a name for the new source instance

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

31

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your CA APM DB

The Metadata Window Width and Latency control time window of metric data which is

extracted to determine the available monitored hosts for selection

Note It is also possible to configure selection to use regexp expressions which are used directly

in the JDBC queries sent to CA APM Depending on whether or not you check the regexp

button you will be presented with a list of Agents(Hosts) to select from or be required to

provide regexps for Agents and Metric selection - see next section

The TimeZone is that of the CA APM server This must be configured to ensure the metric data

is correctly mapped to UTC

Enter the connection information for the CA APM Host

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the CA

APM host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

32

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name

Give this destination a unique name This name will also be used as a prefix for the caapm CSV

files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

33

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from CA APM is based upon selecting the hosts and SWP will extract all

metrics for that host

Note If you had chosen to configure via regexp the list of hosts panel will be empty

You must provide a set of comma separated regexp pairs - one pair per line in the RegExp

panel eg

User

These are regexps for CA APM Agents and Metrics and are used directly in the queries eg

select from metric_data where agent= and metric=User

See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

34

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Note Sometimes depending on the responsiveness of the CA-APM system it may take a few

seconds for the list of hosts to appear Clicking in the panel often helps with refresh

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

35

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

36

Dynatrace Managed

Overview

This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api

Extraction Approach

Pre-requisites

The hostport and REST api connection token for your Dynatrace Managed system (see your

Admin andor Dynatrace API - Authentication)

Mediator creation and configuration

To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new origin

Press lsquo+rsquo on the SOURCES column to add a new source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

37

Now provide the connectivity information for your Dynatrace Managed system

Enter your Dynatrace Managed credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

Dynatrace Managed environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to produce IBM PI CSV files

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

38

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Select the desired Dynatrace Managed metric sources

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

39

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 5: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

v

CONFIGURE DISTRIBUTORS 46 CONFIGURE STREAM 47

ORACLE OEM MEDIATOR 48

OVERVIEW 48 MEDIATOR CREATION AND CONFIGURATION 48 SELECT SOURCE 48 CONNECT TO SOURCE 48 SELECT DESTINATION 49 CONNECT TO DESTINATION 49 SELECT DATA TYPES 50 CONFIGURING COLLECTORS 50 CONFIGURING DISTRIBUTORS 51 SAVE STREAM 51

PIVOTAL CLOUD FOUNDRY MEDIATOR 52

OVERVIEW 52 EXTRACTION APPROACH 52 PRE-REQUISITES 52 OVERALL SETUP 53 MEDIATOR CREATION AND CONFIGURATION 54 CREATE NEW SOURCE 54 CONNECT TO SOURCE 54 SELECT DESTINATION 55 CONNECT TO DESTINATION 56 SELECT DATA TYPES 56 CONFIGURING COLLECTORS 56 CONFIGURE DISTRIBUTORS 58 CONFIGURE STREAM 59 NOZZLE SETUP 61 OPERATION 61

SAP HANA MEDIATOR 63

OVERVIEW 63 EXTRACTION APPROACH 63 SAP HANA PORT DEFAULTS 63 PRE-REQUISITES 63 MEDIATOR CREATION AND CONFIGURATION 63 CREATE NEW SOURCE 63 CONNECT TO SOURCE 64 SELECT DESTINATION 65 CONNECT TO DESTINATION 65 SELECT DATA TYPES 65 CONFIGURING COLLECTORS 66 CONFIGURE DISTRIBUTORS 67 CONFIGURE STREAM 68

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

vi

OPERATIONS 70

STARTING METRIC EXTRACTION 70 STOPPING METRIC EXTRACTION 70 RUNNING SINGLE EXTRACTIONS 70 EDITING AN EXISTING STREAM 72

PREDICTIVE INSIGHTS CSV OUTPUT 74

FILE DETAILS 74 TIMESTAMPS 76

PREDICTIVE INSIGHTS lsquoMODELrsquo FILES 77

GROUP INFORMATION 77 OBTAINING METRIC NAMES FOR PI MEDIATION TOOL 78

PREDICTIVE INSIGHTS REST OUTPUT 80

LOGS 81 CHANGING LOG LEVELS 81

SUPPORT 83

APPENDIX 1 SAP HANA METRICS 84

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

7

Overview

IBM Operations Analytics Predictive Insights analyzes data and learns the normal behavior of a

system through KPI metric data streams This includes data from various sources including

metrics gathered by industry performance and network monitoring solutions A mediation pack

helps users quickly integrate the KPI data streams by providing a default model to integrate a

particular performance manager

IBM Operations Analytics Predictive Insights Mediation Pack Group 2 known in the rest of this

document as the lsquoMediation Packrsquo provides a pre-built model as a starting point to facilitate data

ingestion Predictive Insights creates a performance model that it uses to detect or forecast

behavior outside the modeled range including generating an alarm when an anomalous behavior

occurs Doing so enables organizations to pro-actively avoid system and solution outages before

users are impacted

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

8

The Mediation Pack provides the capability to extract performance data and metrics from the

following sources

bull SAP Hana DB

bull CA APM (Introscope)

bull MicroFocus NNMi

bull AWS Cloudwatch

bull Pivotal Cloud Foundry

bull Dynatrace Managed

bull AppDynamics

bull Oracle OEM

and deliver that data to IBM Operations Analytics Predictive Insights to perform analysis This

document will cover the installation and setup of Mediation Pack for each of the supported source-

data technologies

The figure shows an overall view of the mediation solution in context

Metrics are extracted by Collectors flow through to CSV Distributors which produce CSV files

The CSV files can then be made available to IBM PI for ingestion in the normal way Alternatively

any Collector can be routed to a PI REST Distributor which will deliver data to PI via its REST

interface

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

9

Prerequisite Information

Item Description

Name Mediation Pack Group 2

Predictive Insights supported

versions

v136 and higher Note IBM PI continues to support the CSV

format for ingestion and so earlier versions of PI ie 133 will

also work)

Predictive Insights Information IBM Operations Analytics (demo trial resources)

httpswwwibmcomus-enmarketplaceit-operations-

analytics

IBM Operations Analytics Community (docs videos

mediation packs) httpsdeveloperibmcomitoa

IBM Operations Analytics Predictive Insights

documentation

httpswwwibmcomsupportknowledgecenterSSJQQ3

Solution Name supported

versions

Min version Max Version

CA APM (Introscope) 9x 107

SAP Hana 10 10

AWS Cloudwatch EC2 REST

API 2016-11-

15

Cloudwatch

111

Same as Min

version

MF NNMi 1040x 1040x

Pivotal Cloud Foundry 233 233

Dynatrace Managed Dec 2018 Jun 2019

AppDynamics 41x 459

Oracle OEM 13 13

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

10

Solution Name Information IBM Operations Analytics Predictive Insights Mediation Pack

Group 2

Filename mediation_pack_group2-27-ibmpi-ltbuildNumbergtzip

Supported Operating Systems Linux

RHEL 6x 7x 3264bit

CentOS 6x7x 3264bit

Java Java18+ is required There is a version included in the release

package Note Mediators do NOT run on IBM JVM (the one

delivered with PI at time of writing)

Web Browsers A modern HTML5CSS3Javascript browser Versions of

Chrome Firefox or Safari since 2012 should work Previous

browser versions may not work well or at all

Note Some versions of Internet Explorer (ie 9 or earlier)

have exhibited issues Please consider a more modern

browser to ensure full functionality of the product

Ports Port 7457 (default but configurable) is required by

Mediators

Outbound ports per Mediator Type

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

11

SWP Mediators Overview

The metric data flow is generally from a collection component (a lsquoCollectorrsquo aka lsquoMediatorrsquo)

specific to the data source to an output component (a lsquoDistributorrsquo) Two distributors types are

support 1 CSV - writes metrics to standard IBM Operations Analytics Predictive Insights CSV files

2 REST - posts metric data (as JSON) to the IBM Operations Analytics Predictive Insights REST interface

Whilst each Mediator requires configuration specific to its source all the data flows through to a

corresponding Predictive Insights distributor The configurations approach for each of the

sources will be described in this document

The Mediators as part of the Mediation pack usually will be installed on the same host as

Predictive Insights This makes it easier to deliver the data from the Mediators to Predictive

Insights If the Mediation pack is installed on a different host then mechanisms must be put in

place to ensure that Predictive Insights can get at that CSV data eg shared disk drivesNFS

FTP-type jobs run on a schedule to retrieve files How to establish such mechanisms is beyond

the scope of this document In the case of REST distribution SWP does not have to be co-

located with Predictive Insights

Key steps to setting up Mediators

bull Installation

bull Configure one or more streams which extract the data from the supported sources

bull Configure the lsquoPredictive Insightsrsquo distributor stage for each stream choosing

between CSV and REST output

bull Start the desired metric extraction lsquotasksrsquo for each of the data sources This

should result in Predictive Insights CSV files being produced or JSON data

being posted to the IBM PI REST interface

bull Note Setting upconfiguring IBM PI REST interface is beyond the

scope of this document For details on this please refer to the

corresponding IBM PI documentation

bull In the case of CSV integration configure Predictive Insights for ingestion

using its lsquoMediation Toolrsquo in the normal way Sample Predictive Insights

lsquomodelrsquo files are provided as a starting point The lsquoMediation Toolrsquo is not

used for PI REST integration

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

12

Installing and Configuring

Zip file contents

The file mediation_pack_group2_v27-ibmpi- ltbuildgtzip contains the following

bull streamweaver-ibmpi-27ibmpiltbuildgt-1noarchrpm contains the main SWPMediators executable and associated collateral

bull docs containing the User Guides

o for lsquostreamweaverrsquo (this document)

o for Spl_AppD_Dyna_Ater_MedPack_v15_Revltrevisiongt

bull license containing license files

bull Spl_AppD_Dyna_Ater_MedPack_v15ltbuildgtrpm release with support for

o Dynatrace

o DCRum

o Aternity

o Splunk

o ExtraHop

o CA UIM

o MF APM

o NetIQ AppManager

o Note this package also contains support for AppDynamics Oracle OEM and CA APMIntroscope - these have been deprecated in 1x replaced by the Mediators in the main streamweaver 27 version

Install procedure

New Installation

The following section provides information for installing and configuring the Mediation Pack on

your IBM Operations Analytics Predictive Insights system

Normal lsquorpmrsquo conventions are followed

1 Verify Hardware amp Software Requirements

2 Verify user account privilege requirements

i The installation should be performed as the lsquorootrsquo user or user which can sudo ltgt

with root privileges This is required to create the Linux service

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

13

ii The installer will install as scadmindb2iadm (to match PI)

3 unzip mediation pack file

unzip mediation_pack_group2_v27-ibmpi-ltbuildgtzip

4 Install the bundled JVM

rpm -i --nodeps opstream-jre-18ibmpiltbuildgt-1x86_64rpm

5 Install SWP Mediators Locate the rpm file in the expanded zip directory

rpm -i streamweaver-ibmpi-27ibmpiltbuildgt-

1noarchrpm

After this you should see an indication of successful installation eg

SWP Mediators are installed by default to

optIBMstreamweaver-ibmpi

6 If you used root account exit and return to lsquoscadminrsquo user

Notes

Installing to an alternative location

You can install to a location other than optIBM as follows

sudo rpm -i --relocate optIBM=ltalternativepathgt streamweaver-

ibmpi-27ibmpiltbuildgt-1noarchrpm

Running as non-`scadmin` user

As part of the install if not already present in the system a `scadmin` account and lsquodb2iadm1` group are created for consistency with the default IBM PI installation and related file-ownerships

If you wish to run as a non-`scadmin` user

Change ownership of all the files from down eg

chown -R differentuserdifferentgroup optIBMstreamweaver-ibmpi

Edit the initd startup script

etcinitdstreamweaver-api

Locate and edit the line specifying the DAEMON_USER

[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=scadmin

Change the default `scadmin` to be your desired user eg

[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=differentuser

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

14

Co-existence of SWP-based and Accelerator-based mediators

If you are using the earlier Accelerator-based version along with this SWP-based version

note that they use the same default port setting of 7457 In this situation you should change the

default for SWP to something else You can do this by locating and editing the installed config

file in

optIBMstreamweaver-ibmpi confswpconf

and locate uncomment and change the line with port = 7457 to some other port

Settings for the Core Services services

Settings for the UI Service

ui

How the UI Services binds to a socket

connection

Hostname or IP Address to listen on The default

0000 causes

the REST API to listen on all interfaces

host = 0000

The port number to listen on

port = 7457

to some other free port

Also By default the UI listens on 7457 and the REST API listens on 7456 The UI assumes that the REST port is 1 less than the UI port Having made the above config change the following config entries are required and must be explicitly added to swpconf at the end of the file

httpport = ltrest-port-numbergt must be one less than UI port this defines the

port that REST API listens on lagomservicesswpallinone = httplocalhostltrest-port-

number server-side clients of REST service use this to locate it NOTE quotes are important here

Data directories

If you wish to relocate the various working directories (eg logs data) edit the swpconf file and modify the relevant entries in the ldquopathsrsquo section

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

15

Installation users

SWPMediators is installed by root as usergroup scadmindb2iadm1 to align the default PI usergroups

Starting SWP system

SWP contains Kafka Cassandra and the core They are all run under the lsquoswprsquo service

bull As lsquorootrsquo execute

service streamweaver-ibmpi start

In the usual way running lsquostatusrsquo on the service should give an indication that the main

SWP demon is operational

service streamweaver-ibmpi status

Once the SWP demon is running you use the SWP command line utility to monitor the

status of the child processes

Use lsquoswprsquo to observe the status of the system

bull optIBMstreamweaver-ibmpibinswp status

You should see a response similar to the following

In normal operation the five processes should be in Running state

You can also run the status command in a loop (like Linux `watch`) eg

bull optIBMstreamweaver-ibmpibinswp status -w 3

Stopping SWP system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

16

service streamweaver-ibmpi stop

Upgrade installation

service streamweaver-ibmpi stop

rpm -e streamweaver-ibmpi

Install new version as described above

Deployed directory contents

The main directories are created as part of the installation

optIBMstreamweaver-ibmpi

jre

The bundled JRE

bin

lsquoswpdrsquo all-in-one executable and lsquoswprsquo command line

content-packs

lsquoPI mediation lsquopa modelrsquo files and Pivotal CF lsquonozzlersquo components

lib

Library files (jars)

logs

Log files useful for monitoring and debugging system behavior

conf

Configuration files to control SWP

services

Cassandra Kafka and swpallinone (core SWP services)

Configuration files to control streamweaver

dataswp

Extracted metric data is stored under here

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

17

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

18

Accessing the Mediation system

Once installed you can interact with the Mediation system via web-browser By default the

Mediation system listens on port 7457 so for example point your web-browser to

httpltservergt7457

Weaver Pressing lsquoWEAVERrsquo brings you back to the above screen You select the lsquo+rsquo and a list

of source options will appear Any 3rd party tool listed can be selected to begin the configuration process In a similar fashion Destinations can also be configured by selecting the plus icon in the Destinations section

Dashboard Pressing lsquoDASHBOARDrsquo while cause latency measurements (the duration of last collection for each destination mediator) for each stream to be presented

Configuration Pressing lsquoDASHBOARDrsquo shows configuration information for the platform

including particularly log levels

System Logs Pressing lsquoSYSTEM LOGSrsquo presents the latest log entries (the full set of logs and be viewed in the relevant log directory which is described later in this document)

In the subsequent Mediator-specific sections we will give specific examples of

Mediator-gtDistributor Each example will specify a specific kind of distributor ie CSV or REST

You are free to use either distributor in each case

The initial step in each configuration case is to press lsquo+rsquo on the SOURCES and then you should

be presented with a list of available Mediator source types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

19

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

20

AppDynamics Mediator

Overview

Extraction Approach

Metric data is extracted using the AppDynamics REST API

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

AppD Port Defaults

REST 443

Mediator creation and configuration

To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your AppDynamics source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

21

Select Destination

Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single

file stream emerges from each Destination and we donrsquot generally want to mix data from

different Collector instances

Give this destination a unique name This name will also be used as a prefix for the

AppDynamics CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

22

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from AppDynamics is based upon selecting Applications Tiers and metric

paths SWP will extract all metrics for the supplied paths

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

23

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)

when data for the next time interval arrives

You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open

for the specified number of periods (as defined by the Maxium Time Window Per File) This is

useful when you want to wait for late data The trade-off is that the files are only made available

to PI after that lag period - thereby increasing end to end latency

For example if you had 5min files and set the max lag to 2 yoursquod have current file open and

two older files leading to up to 15 min latency

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

24

AWS Cloudwatch Mediator

Overview

This mediator extracts Metric data from AWS Cloudwatch system

Extraction Approach

Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference

Pre-requisites

You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users

The Endpoint regions are shown in the following table For the subsequent SWP configuration

you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account

has access to metric data

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

25

T

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

26

Mediator creation and configuration

To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the Source column to add a new origin

Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for AWS Cloudwatch account

Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least

one Region Multiple regions can be entered by separating each with a comma

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the AWS

CloudWatch environment)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

27

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

28

Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics

in those categories SWP will extract all metrics for those selected metric tables For IBM PI

only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are

selected

The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide

convenient resource groupings If you want to scope your metric extraction to particular tagged

instances specify those tags here as comma separated tagname=tagvalue pairs

For more details on AWS tagging see the AWS documentation

The optimum settings for the other configuration values depend on your AWS accountdata

availability etc The defaults should be sufficient to get you operational

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

29

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

30

CA APM (Introscope) Mediator

Overview

CA APM was formerly known as Wily Introscope

Metric extraction from CA APM is based upon configuring the system to extract data for specific

Hosts Metric data will be collected for all Agents and Processes on each selected host

Extraction Approach

Metric data is extracted using a JDBC interface

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

CA APM Port Defaults

JDBC 5001

Note

The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used

Mediator creation and configuration

To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

and then press lsquo+rsquo on the Sources column to add a new sources

Provide a name for the new source instance

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

31

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your CA APM DB

The Metadata Window Width and Latency control time window of metric data which is

extracted to determine the available monitored hosts for selection

Note It is also possible to configure selection to use regexp expressions which are used directly

in the JDBC queries sent to CA APM Depending on whether or not you check the regexp

button you will be presented with a list of Agents(Hosts) to select from or be required to

provide regexps for Agents and Metric selection - see next section

The TimeZone is that of the CA APM server This must be configured to ensure the metric data

is correctly mapped to UTC

Enter the connection information for the CA APM Host

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the CA

APM host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

32

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name

Give this destination a unique name This name will also be used as a prefix for the caapm CSV

files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

33

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from CA APM is based upon selecting the hosts and SWP will extract all

metrics for that host

Note If you had chosen to configure via regexp the list of hosts panel will be empty

You must provide a set of comma separated regexp pairs - one pair per line in the RegExp

panel eg

User

These are regexps for CA APM Agents and Metrics and are used directly in the queries eg

select from metric_data where agent= and metric=User

See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

34

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Note Sometimes depending on the responsiveness of the CA-APM system it may take a few

seconds for the list of hosts to appear Clicking in the panel often helps with refresh

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

35

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

36

Dynatrace Managed

Overview

This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api

Extraction Approach

Pre-requisites

The hostport and REST api connection token for your Dynatrace Managed system (see your

Admin andor Dynatrace API - Authentication)

Mediator creation and configuration

To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new origin

Press lsquo+rsquo on the SOURCES column to add a new source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

37

Now provide the connectivity information for your Dynatrace Managed system

Enter your Dynatrace Managed credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

Dynatrace Managed environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to produce IBM PI CSV files

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

38

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Select the desired Dynatrace Managed metric sources

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

39

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 6: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

vi

OPERATIONS 70

STARTING METRIC EXTRACTION 70 STOPPING METRIC EXTRACTION 70 RUNNING SINGLE EXTRACTIONS 70 EDITING AN EXISTING STREAM 72

PREDICTIVE INSIGHTS CSV OUTPUT 74

FILE DETAILS 74 TIMESTAMPS 76

PREDICTIVE INSIGHTS lsquoMODELrsquo FILES 77

GROUP INFORMATION 77 OBTAINING METRIC NAMES FOR PI MEDIATION TOOL 78

PREDICTIVE INSIGHTS REST OUTPUT 80

LOGS 81 CHANGING LOG LEVELS 81

SUPPORT 83

APPENDIX 1 SAP HANA METRICS 84

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

7

Overview

IBM Operations Analytics Predictive Insights analyzes data and learns the normal behavior of a

system through KPI metric data streams This includes data from various sources including

metrics gathered by industry performance and network monitoring solutions A mediation pack

helps users quickly integrate the KPI data streams by providing a default model to integrate a

particular performance manager

IBM Operations Analytics Predictive Insights Mediation Pack Group 2 known in the rest of this

document as the lsquoMediation Packrsquo provides a pre-built model as a starting point to facilitate data

ingestion Predictive Insights creates a performance model that it uses to detect or forecast

behavior outside the modeled range including generating an alarm when an anomalous behavior

occurs Doing so enables organizations to pro-actively avoid system and solution outages before

users are impacted

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

8

The Mediation Pack provides the capability to extract performance data and metrics from the

following sources

bull SAP Hana DB

bull CA APM (Introscope)

bull MicroFocus NNMi

bull AWS Cloudwatch

bull Pivotal Cloud Foundry

bull Dynatrace Managed

bull AppDynamics

bull Oracle OEM

and deliver that data to IBM Operations Analytics Predictive Insights to perform analysis This

document will cover the installation and setup of Mediation Pack for each of the supported source-

data technologies

The figure shows an overall view of the mediation solution in context

Metrics are extracted by Collectors flow through to CSV Distributors which produce CSV files

The CSV files can then be made available to IBM PI for ingestion in the normal way Alternatively

any Collector can be routed to a PI REST Distributor which will deliver data to PI via its REST

interface

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

9

Prerequisite Information

Item Description

Name Mediation Pack Group 2

Predictive Insights supported

versions

v136 and higher Note IBM PI continues to support the CSV

format for ingestion and so earlier versions of PI ie 133 will

also work)

Predictive Insights Information IBM Operations Analytics (demo trial resources)

httpswwwibmcomus-enmarketplaceit-operations-

analytics

IBM Operations Analytics Community (docs videos

mediation packs) httpsdeveloperibmcomitoa

IBM Operations Analytics Predictive Insights

documentation

httpswwwibmcomsupportknowledgecenterSSJQQ3

Solution Name supported

versions

Min version Max Version

CA APM (Introscope) 9x 107

SAP Hana 10 10

AWS Cloudwatch EC2 REST

API 2016-11-

15

Cloudwatch

111

Same as Min

version

MF NNMi 1040x 1040x

Pivotal Cloud Foundry 233 233

Dynatrace Managed Dec 2018 Jun 2019

AppDynamics 41x 459

Oracle OEM 13 13

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

10

Solution Name Information IBM Operations Analytics Predictive Insights Mediation Pack

Group 2

Filename mediation_pack_group2-27-ibmpi-ltbuildNumbergtzip

Supported Operating Systems Linux

RHEL 6x 7x 3264bit

CentOS 6x7x 3264bit

Java Java18+ is required There is a version included in the release

package Note Mediators do NOT run on IBM JVM (the one

delivered with PI at time of writing)

Web Browsers A modern HTML5CSS3Javascript browser Versions of

Chrome Firefox or Safari since 2012 should work Previous

browser versions may not work well or at all

Note Some versions of Internet Explorer (ie 9 or earlier)

have exhibited issues Please consider a more modern

browser to ensure full functionality of the product

Ports Port 7457 (default but configurable) is required by

Mediators

Outbound ports per Mediator Type

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

11

SWP Mediators Overview

The metric data flow is generally from a collection component (a lsquoCollectorrsquo aka lsquoMediatorrsquo)

specific to the data source to an output component (a lsquoDistributorrsquo) Two distributors types are

support 1 CSV - writes metrics to standard IBM Operations Analytics Predictive Insights CSV files

2 REST - posts metric data (as JSON) to the IBM Operations Analytics Predictive Insights REST interface

Whilst each Mediator requires configuration specific to its source all the data flows through to a

corresponding Predictive Insights distributor The configurations approach for each of the

sources will be described in this document

The Mediators as part of the Mediation pack usually will be installed on the same host as

Predictive Insights This makes it easier to deliver the data from the Mediators to Predictive

Insights If the Mediation pack is installed on a different host then mechanisms must be put in

place to ensure that Predictive Insights can get at that CSV data eg shared disk drivesNFS

FTP-type jobs run on a schedule to retrieve files How to establish such mechanisms is beyond

the scope of this document In the case of REST distribution SWP does not have to be co-

located with Predictive Insights

Key steps to setting up Mediators

bull Installation

bull Configure one or more streams which extract the data from the supported sources

bull Configure the lsquoPredictive Insightsrsquo distributor stage for each stream choosing

between CSV and REST output

bull Start the desired metric extraction lsquotasksrsquo for each of the data sources This

should result in Predictive Insights CSV files being produced or JSON data

being posted to the IBM PI REST interface

bull Note Setting upconfiguring IBM PI REST interface is beyond the

scope of this document For details on this please refer to the

corresponding IBM PI documentation

bull In the case of CSV integration configure Predictive Insights for ingestion

using its lsquoMediation Toolrsquo in the normal way Sample Predictive Insights

lsquomodelrsquo files are provided as a starting point The lsquoMediation Toolrsquo is not

used for PI REST integration

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

12

Installing and Configuring

Zip file contents

The file mediation_pack_group2_v27-ibmpi- ltbuildgtzip contains the following

bull streamweaver-ibmpi-27ibmpiltbuildgt-1noarchrpm contains the main SWPMediators executable and associated collateral

bull docs containing the User Guides

o for lsquostreamweaverrsquo (this document)

o for Spl_AppD_Dyna_Ater_MedPack_v15_Revltrevisiongt

bull license containing license files

bull Spl_AppD_Dyna_Ater_MedPack_v15ltbuildgtrpm release with support for

o Dynatrace

o DCRum

o Aternity

o Splunk

o ExtraHop

o CA UIM

o MF APM

o NetIQ AppManager

o Note this package also contains support for AppDynamics Oracle OEM and CA APMIntroscope - these have been deprecated in 1x replaced by the Mediators in the main streamweaver 27 version

Install procedure

New Installation

The following section provides information for installing and configuring the Mediation Pack on

your IBM Operations Analytics Predictive Insights system

Normal lsquorpmrsquo conventions are followed

1 Verify Hardware amp Software Requirements

2 Verify user account privilege requirements

i The installation should be performed as the lsquorootrsquo user or user which can sudo ltgt

with root privileges This is required to create the Linux service

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

13

ii The installer will install as scadmindb2iadm (to match PI)

3 unzip mediation pack file

unzip mediation_pack_group2_v27-ibmpi-ltbuildgtzip

4 Install the bundled JVM

rpm -i --nodeps opstream-jre-18ibmpiltbuildgt-1x86_64rpm

5 Install SWP Mediators Locate the rpm file in the expanded zip directory

rpm -i streamweaver-ibmpi-27ibmpiltbuildgt-

1noarchrpm

After this you should see an indication of successful installation eg

SWP Mediators are installed by default to

optIBMstreamweaver-ibmpi

6 If you used root account exit and return to lsquoscadminrsquo user

Notes

Installing to an alternative location

You can install to a location other than optIBM as follows

sudo rpm -i --relocate optIBM=ltalternativepathgt streamweaver-

ibmpi-27ibmpiltbuildgt-1noarchrpm

Running as non-`scadmin` user

As part of the install if not already present in the system a `scadmin` account and lsquodb2iadm1` group are created for consistency with the default IBM PI installation and related file-ownerships

If you wish to run as a non-`scadmin` user

Change ownership of all the files from down eg

chown -R differentuserdifferentgroup optIBMstreamweaver-ibmpi

Edit the initd startup script

etcinitdstreamweaver-api

Locate and edit the line specifying the DAEMON_USER

[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=scadmin

Change the default `scadmin` to be your desired user eg

[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=differentuser

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

14

Co-existence of SWP-based and Accelerator-based mediators

If you are using the earlier Accelerator-based version along with this SWP-based version

note that they use the same default port setting of 7457 In this situation you should change the

default for SWP to something else You can do this by locating and editing the installed config

file in

optIBMstreamweaver-ibmpi confswpconf

and locate uncomment and change the line with port = 7457 to some other port

Settings for the Core Services services

Settings for the UI Service

ui

How the UI Services binds to a socket

connection

Hostname or IP Address to listen on The default

0000 causes

the REST API to listen on all interfaces

host = 0000

The port number to listen on

port = 7457

to some other free port

Also By default the UI listens on 7457 and the REST API listens on 7456 The UI assumes that the REST port is 1 less than the UI port Having made the above config change the following config entries are required and must be explicitly added to swpconf at the end of the file

httpport = ltrest-port-numbergt must be one less than UI port this defines the

port that REST API listens on lagomservicesswpallinone = httplocalhostltrest-port-

number server-side clients of REST service use this to locate it NOTE quotes are important here

Data directories

If you wish to relocate the various working directories (eg logs data) edit the swpconf file and modify the relevant entries in the ldquopathsrsquo section

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

15

Installation users

SWPMediators is installed by root as usergroup scadmindb2iadm1 to align the default PI usergroups

Starting SWP system

SWP contains Kafka Cassandra and the core They are all run under the lsquoswprsquo service

bull As lsquorootrsquo execute

service streamweaver-ibmpi start

In the usual way running lsquostatusrsquo on the service should give an indication that the main

SWP demon is operational

service streamweaver-ibmpi status

Once the SWP demon is running you use the SWP command line utility to monitor the

status of the child processes

Use lsquoswprsquo to observe the status of the system

bull optIBMstreamweaver-ibmpibinswp status

You should see a response similar to the following

In normal operation the five processes should be in Running state

You can also run the status command in a loop (like Linux `watch`) eg

bull optIBMstreamweaver-ibmpibinswp status -w 3

Stopping SWP system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

16

service streamweaver-ibmpi stop

Upgrade installation

service streamweaver-ibmpi stop

rpm -e streamweaver-ibmpi

Install new version as described above

Deployed directory contents

The main directories are created as part of the installation

optIBMstreamweaver-ibmpi

jre

The bundled JRE

bin

lsquoswpdrsquo all-in-one executable and lsquoswprsquo command line

content-packs

lsquoPI mediation lsquopa modelrsquo files and Pivotal CF lsquonozzlersquo components

lib

Library files (jars)

logs

Log files useful for monitoring and debugging system behavior

conf

Configuration files to control SWP

services

Cassandra Kafka and swpallinone (core SWP services)

Configuration files to control streamweaver

dataswp

Extracted metric data is stored under here

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

17

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

18

Accessing the Mediation system

Once installed you can interact with the Mediation system via web-browser By default the

Mediation system listens on port 7457 so for example point your web-browser to

httpltservergt7457

Weaver Pressing lsquoWEAVERrsquo brings you back to the above screen You select the lsquo+rsquo and a list

of source options will appear Any 3rd party tool listed can be selected to begin the configuration process In a similar fashion Destinations can also be configured by selecting the plus icon in the Destinations section

Dashboard Pressing lsquoDASHBOARDrsquo while cause latency measurements (the duration of last collection for each destination mediator) for each stream to be presented

Configuration Pressing lsquoDASHBOARDrsquo shows configuration information for the platform

including particularly log levels

System Logs Pressing lsquoSYSTEM LOGSrsquo presents the latest log entries (the full set of logs and be viewed in the relevant log directory which is described later in this document)

In the subsequent Mediator-specific sections we will give specific examples of

Mediator-gtDistributor Each example will specify a specific kind of distributor ie CSV or REST

You are free to use either distributor in each case

The initial step in each configuration case is to press lsquo+rsquo on the SOURCES and then you should

be presented with a list of available Mediator source types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

19

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

20

AppDynamics Mediator

Overview

Extraction Approach

Metric data is extracted using the AppDynamics REST API

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

AppD Port Defaults

REST 443

Mediator creation and configuration

To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your AppDynamics source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

21

Select Destination

Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single

file stream emerges from each Destination and we donrsquot generally want to mix data from

different Collector instances

Give this destination a unique name This name will also be used as a prefix for the

AppDynamics CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

22

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from AppDynamics is based upon selecting Applications Tiers and metric

paths SWP will extract all metrics for the supplied paths

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

23

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)

when data for the next time interval arrives

You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open

for the specified number of periods (as defined by the Maxium Time Window Per File) This is

useful when you want to wait for late data The trade-off is that the files are only made available

to PI after that lag period - thereby increasing end to end latency

For example if you had 5min files and set the max lag to 2 yoursquod have current file open and

two older files leading to up to 15 min latency

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

24

AWS Cloudwatch Mediator

Overview

This mediator extracts Metric data from AWS Cloudwatch system

Extraction Approach

Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference

Pre-requisites

You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users

The Endpoint regions are shown in the following table For the subsequent SWP configuration

you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account

has access to metric data

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

25

T

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

26

Mediator creation and configuration

To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the Source column to add a new origin

Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for AWS Cloudwatch account

Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least

one Region Multiple regions can be entered by separating each with a comma

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the AWS

CloudWatch environment)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

27

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

28

Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics

in those categories SWP will extract all metrics for those selected metric tables For IBM PI

only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are

selected

The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide

convenient resource groupings If you want to scope your metric extraction to particular tagged

instances specify those tags here as comma separated tagname=tagvalue pairs

For more details on AWS tagging see the AWS documentation

The optimum settings for the other configuration values depend on your AWS accountdata

availability etc The defaults should be sufficient to get you operational

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

29

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

30

CA APM (Introscope) Mediator

Overview

CA APM was formerly known as Wily Introscope

Metric extraction from CA APM is based upon configuring the system to extract data for specific

Hosts Metric data will be collected for all Agents and Processes on each selected host

Extraction Approach

Metric data is extracted using a JDBC interface

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

CA APM Port Defaults

JDBC 5001

Note

The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used

Mediator creation and configuration

To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

and then press lsquo+rsquo on the Sources column to add a new sources

Provide a name for the new source instance

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

31

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your CA APM DB

The Metadata Window Width and Latency control time window of metric data which is

extracted to determine the available monitored hosts for selection

Note It is also possible to configure selection to use regexp expressions which are used directly

in the JDBC queries sent to CA APM Depending on whether or not you check the regexp

button you will be presented with a list of Agents(Hosts) to select from or be required to

provide regexps for Agents and Metric selection - see next section

The TimeZone is that of the CA APM server This must be configured to ensure the metric data

is correctly mapped to UTC

Enter the connection information for the CA APM Host

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the CA

APM host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

32

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name

Give this destination a unique name This name will also be used as a prefix for the caapm CSV

files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

33

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from CA APM is based upon selecting the hosts and SWP will extract all

metrics for that host

Note If you had chosen to configure via regexp the list of hosts panel will be empty

You must provide a set of comma separated regexp pairs - one pair per line in the RegExp

panel eg

User

These are regexps for CA APM Agents and Metrics and are used directly in the queries eg

select from metric_data where agent= and metric=User

See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

34

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Note Sometimes depending on the responsiveness of the CA-APM system it may take a few

seconds for the list of hosts to appear Clicking in the panel often helps with refresh

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

35

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

36

Dynatrace Managed

Overview

This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api

Extraction Approach

Pre-requisites

The hostport and REST api connection token for your Dynatrace Managed system (see your

Admin andor Dynatrace API - Authentication)

Mediator creation and configuration

To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new origin

Press lsquo+rsquo on the SOURCES column to add a new source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

37

Now provide the connectivity information for your Dynatrace Managed system

Enter your Dynatrace Managed credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

Dynatrace Managed environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to produce IBM PI CSV files

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

38

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Select the desired Dynatrace Managed metric sources

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

39

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 7: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

7

Overview

IBM Operations Analytics Predictive Insights analyzes data and learns the normal behavior of a

system through KPI metric data streams This includes data from various sources including

metrics gathered by industry performance and network monitoring solutions A mediation pack

helps users quickly integrate the KPI data streams by providing a default model to integrate a

particular performance manager

IBM Operations Analytics Predictive Insights Mediation Pack Group 2 known in the rest of this

document as the lsquoMediation Packrsquo provides a pre-built model as a starting point to facilitate data

ingestion Predictive Insights creates a performance model that it uses to detect or forecast

behavior outside the modeled range including generating an alarm when an anomalous behavior

occurs Doing so enables organizations to pro-actively avoid system and solution outages before

users are impacted

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

8

The Mediation Pack provides the capability to extract performance data and metrics from the

following sources

bull SAP Hana DB

bull CA APM (Introscope)

bull MicroFocus NNMi

bull AWS Cloudwatch

bull Pivotal Cloud Foundry

bull Dynatrace Managed

bull AppDynamics

bull Oracle OEM

and deliver that data to IBM Operations Analytics Predictive Insights to perform analysis This

document will cover the installation and setup of Mediation Pack for each of the supported source-

data technologies

The figure shows an overall view of the mediation solution in context

Metrics are extracted by Collectors flow through to CSV Distributors which produce CSV files

The CSV files can then be made available to IBM PI for ingestion in the normal way Alternatively

any Collector can be routed to a PI REST Distributor which will deliver data to PI via its REST

interface

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

9

Prerequisite Information

Item Description

Name Mediation Pack Group 2

Predictive Insights supported

versions

v136 and higher Note IBM PI continues to support the CSV

format for ingestion and so earlier versions of PI ie 133 will

also work)

Predictive Insights Information IBM Operations Analytics (demo trial resources)

httpswwwibmcomus-enmarketplaceit-operations-

analytics

IBM Operations Analytics Community (docs videos

mediation packs) httpsdeveloperibmcomitoa

IBM Operations Analytics Predictive Insights

documentation

httpswwwibmcomsupportknowledgecenterSSJQQ3

Solution Name supported

versions

Min version Max Version

CA APM (Introscope) 9x 107

SAP Hana 10 10

AWS Cloudwatch EC2 REST

API 2016-11-

15

Cloudwatch

111

Same as Min

version

MF NNMi 1040x 1040x

Pivotal Cloud Foundry 233 233

Dynatrace Managed Dec 2018 Jun 2019

AppDynamics 41x 459

Oracle OEM 13 13

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

10

Solution Name Information IBM Operations Analytics Predictive Insights Mediation Pack

Group 2

Filename mediation_pack_group2-27-ibmpi-ltbuildNumbergtzip

Supported Operating Systems Linux

RHEL 6x 7x 3264bit

CentOS 6x7x 3264bit

Java Java18+ is required There is a version included in the release

package Note Mediators do NOT run on IBM JVM (the one

delivered with PI at time of writing)

Web Browsers A modern HTML5CSS3Javascript browser Versions of

Chrome Firefox or Safari since 2012 should work Previous

browser versions may not work well or at all

Note Some versions of Internet Explorer (ie 9 or earlier)

have exhibited issues Please consider a more modern

browser to ensure full functionality of the product

Ports Port 7457 (default but configurable) is required by

Mediators

Outbound ports per Mediator Type

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

11

SWP Mediators Overview

The metric data flow is generally from a collection component (a lsquoCollectorrsquo aka lsquoMediatorrsquo)

specific to the data source to an output component (a lsquoDistributorrsquo) Two distributors types are

support 1 CSV - writes metrics to standard IBM Operations Analytics Predictive Insights CSV files

2 REST - posts metric data (as JSON) to the IBM Operations Analytics Predictive Insights REST interface

Whilst each Mediator requires configuration specific to its source all the data flows through to a

corresponding Predictive Insights distributor The configurations approach for each of the

sources will be described in this document

The Mediators as part of the Mediation pack usually will be installed on the same host as

Predictive Insights This makes it easier to deliver the data from the Mediators to Predictive

Insights If the Mediation pack is installed on a different host then mechanisms must be put in

place to ensure that Predictive Insights can get at that CSV data eg shared disk drivesNFS

FTP-type jobs run on a schedule to retrieve files How to establish such mechanisms is beyond

the scope of this document In the case of REST distribution SWP does not have to be co-

located with Predictive Insights

Key steps to setting up Mediators

bull Installation

bull Configure one or more streams which extract the data from the supported sources

bull Configure the lsquoPredictive Insightsrsquo distributor stage for each stream choosing

between CSV and REST output

bull Start the desired metric extraction lsquotasksrsquo for each of the data sources This

should result in Predictive Insights CSV files being produced or JSON data

being posted to the IBM PI REST interface

bull Note Setting upconfiguring IBM PI REST interface is beyond the

scope of this document For details on this please refer to the

corresponding IBM PI documentation

bull In the case of CSV integration configure Predictive Insights for ingestion

using its lsquoMediation Toolrsquo in the normal way Sample Predictive Insights

lsquomodelrsquo files are provided as a starting point The lsquoMediation Toolrsquo is not

used for PI REST integration

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

12

Installing and Configuring

Zip file contents

The file mediation_pack_group2_v27-ibmpi- ltbuildgtzip contains the following

bull streamweaver-ibmpi-27ibmpiltbuildgt-1noarchrpm contains the main SWPMediators executable and associated collateral

bull docs containing the User Guides

o for lsquostreamweaverrsquo (this document)

o for Spl_AppD_Dyna_Ater_MedPack_v15_Revltrevisiongt

bull license containing license files

bull Spl_AppD_Dyna_Ater_MedPack_v15ltbuildgtrpm release with support for

o Dynatrace

o DCRum

o Aternity

o Splunk

o ExtraHop

o CA UIM

o MF APM

o NetIQ AppManager

o Note this package also contains support for AppDynamics Oracle OEM and CA APMIntroscope - these have been deprecated in 1x replaced by the Mediators in the main streamweaver 27 version

Install procedure

New Installation

The following section provides information for installing and configuring the Mediation Pack on

your IBM Operations Analytics Predictive Insights system

Normal lsquorpmrsquo conventions are followed

1 Verify Hardware amp Software Requirements

2 Verify user account privilege requirements

i The installation should be performed as the lsquorootrsquo user or user which can sudo ltgt

with root privileges This is required to create the Linux service

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

13

ii The installer will install as scadmindb2iadm (to match PI)

3 unzip mediation pack file

unzip mediation_pack_group2_v27-ibmpi-ltbuildgtzip

4 Install the bundled JVM

rpm -i --nodeps opstream-jre-18ibmpiltbuildgt-1x86_64rpm

5 Install SWP Mediators Locate the rpm file in the expanded zip directory

rpm -i streamweaver-ibmpi-27ibmpiltbuildgt-

1noarchrpm

After this you should see an indication of successful installation eg

SWP Mediators are installed by default to

optIBMstreamweaver-ibmpi

6 If you used root account exit and return to lsquoscadminrsquo user

Notes

Installing to an alternative location

You can install to a location other than optIBM as follows

sudo rpm -i --relocate optIBM=ltalternativepathgt streamweaver-

ibmpi-27ibmpiltbuildgt-1noarchrpm

Running as non-`scadmin` user

As part of the install if not already present in the system a `scadmin` account and lsquodb2iadm1` group are created for consistency with the default IBM PI installation and related file-ownerships

If you wish to run as a non-`scadmin` user

Change ownership of all the files from down eg

chown -R differentuserdifferentgroup optIBMstreamweaver-ibmpi

Edit the initd startup script

etcinitdstreamweaver-api

Locate and edit the line specifying the DAEMON_USER

[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=scadmin

Change the default `scadmin` to be your desired user eg

[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=differentuser

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

14

Co-existence of SWP-based and Accelerator-based mediators

If you are using the earlier Accelerator-based version along with this SWP-based version

note that they use the same default port setting of 7457 In this situation you should change the

default for SWP to something else You can do this by locating and editing the installed config

file in

optIBMstreamweaver-ibmpi confswpconf

and locate uncomment and change the line with port = 7457 to some other port

Settings for the Core Services services

Settings for the UI Service

ui

How the UI Services binds to a socket

connection

Hostname or IP Address to listen on The default

0000 causes

the REST API to listen on all interfaces

host = 0000

The port number to listen on

port = 7457

to some other free port

Also By default the UI listens on 7457 and the REST API listens on 7456 The UI assumes that the REST port is 1 less than the UI port Having made the above config change the following config entries are required and must be explicitly added to swpconf at the end of the file

httpport = ltrest-port-numbergt must be one less than UI port this defines the

port that REST API listens on lagomservicesswpallinone = httplocalhostltrest-port-

number server-side clients of REST service use this to locate it NOTE quotes are important here

Data directories

If you wish to relocate the various working directories (eg logs data) edit the swpconf file and modify the relevant entries in the ldquopathsrsquo section

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

15

Installation users

SWPMediators is installed by root as usergroup scadmindb2iadm1 to align the default PI usergroups

Starting SWP system

SWP contains Kafka Cassandra and the core They are all run under the lsquoswprsquo service

bull As lsquorootrsquo execute

service streamweaver-ibmpi start

In the usual way running lsquostatusrsquo on the service should give an indication that the main

SWP demon is operational

service streamweaver-ibmpi status

Once the SWP demon is running you use the SWP command line utility to monitor the

status of the child processes

Use lsquoswprsquo to observe the status of the system

bull optIBMstreamweaver-ibmpibinswp status

You should see a response similar to the following

In normal operation the five processes should be in Running state

You can also run the status command in a loop (like Linux `watch`) eg

bull optIBMstreamweaver-ibmpibinswp status -w 3

Stopping SWP system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

16

service streamweaver-ibmpi stop

Upgrade installation

service streamweaver-ibmpi stop

rpm -e streamweaver-ibmpi

Install new version as described above

Deployed directory contents

The main directories are created as part of the installation

optIBMstreamweaver-ibmpi

jre

The bundled JRE

bin

lsquoswpdrsquo all-in-one executable and lsquoswprsquo command line

content-packs

lsquoPI mediation lsquopa modelrsquo files and Pivotal CF lsquonozzlersquo components

lib

Library files (jars)

logs

Log files useful for monitoring and debugging system behavior

conf

Configuration files to control SWP

services

Cassandra Kafka and swpallinone (core SWP services)

Configuration files to control streamweaver

dataswp

Extracted metric data is stored under here

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

17

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

18

Accessing the Mediation system

Once installed you can interact with the Mediation system via web-browser By default the

Mediation system listens on port 7457 so for example point your web-browser to

httpltservergt7457

Weaver Pressing lsquoWEAVERrsquo brings you back to the above screen You select the lsquo+rsquo and a list

of source options will appear Any 3rd party tool listed can be selected to begin the configuration process In a similar fashion Destinations can also be configured by selecting the plus icon in the Destinations section

Dashboard Pressing lsquoDASHBOARDrsquo while cause latency measurements (the duration of last collection for each destination mediator) for each stream to be presented

Configuration Pressing lsquoDASHBOARDrsquo shows configuration information for the platform

including particularly log levels

System Logs Pressing lsquoSYSTEM LOGSrsquo presents the latest log entries (the full set of logs and be viewed in the relevant log directory which is described later in this document)

In the subsequent Mediator-specific sections we will give specific examples of

Mediator-gtDistributor Each example will specify a specific kind of distributor ie CSV or REST

You are free to use either distributor in each case

The initial step in each configuration case is to press lsquo+rsquo on the SOURCES and then you should

be presented with a list of available Mediator source types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

19

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

20

AppDynamics Mediator

Overview

Extraction Approach

Metric data is extracted using the AppDynamics REST API

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

AppD Port Defaults

REST 443

Mediator creation and configuration

To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your AppDynamics source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

21

Select Destination

Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single

file stream emerges from each Destination and we donrsquot generally want to mix data from

different Collector instances

Give this destination a unique name This name will also be used as a prefix for the

AppDynamics CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

22

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from AppDynamics is based upon selecting Applications Tiers and metric

paths SWP will extract all metrics for the supplied paths

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

23

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)

when data for the next time interval arrives

You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open

for the specified number of periods (as defined by the Maxium Time Window Per File) This is

useful when you want to wait for late data The trade-off is that the files are only made available

to PI after that lag period - thereby increasing end to end latency

For example if you had 5min files and set the max lag to 2 yoursquod have current file open and

two older files leading to up to 15 min latency

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

24

AWS Cloudwatch Mediator

Overview

This mediator extracts Metric data from AWS Cloudwatch system

Extraction Approach

Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference

Pre-requisites

You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users

The Endpoint regions are shown in the following table For the subsequent SWP configuration

you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account

has access to metric data

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

25

T

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

26

Mediator creation and configuration

To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the Source column to add a new origin

Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for AWS Cloudwatch account

Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least

one Region Multiple regions can be entered by separating each with a comma

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the AWS

CloudWatch environment)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

27

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

28

Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics

in those categories SWP will extract all metrics for those selected metric tables For IBM PI

only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are

selected

The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide

convenient resource groupings If you want to scope your metric extraction to particular tagged

instances specify those tags here as comma separated tagname=tagvalue pairs

For more details on AWS tagging see the AWS documentation

The optimum settings for the other configuration values depend on your AWS accountdata

availability etc The defaults should be sufficient to get you operational

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

29

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

30

CA APM (Introscope) Mediator

Overview

CA APM was formerly known as Wily Introscope

Metric extraction from CA APM is based upon configuring the system to extract data for specific

Hosts Metric data will be collected for all Agents and Processes on each selected host

Extraction Approach

Metric data is extracted using a JDBC interface

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

CA APM Port Defaults

JDBC 5001

Note

The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used

Mediator creation and configuration

To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

and then press lsquo+rsquo on the Sources column to add a new sources

Provide a name for the new source instance

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

31

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your CA APM DB

The Metadata Window Width and Latency control time window of metric data which is

extracted to determine the available monitored hosts for selection

Note It is also possible to configure selection to use regexp expressions which are used directly

in the JDBC queries sent to CA APM Depending on whether or not you check the regexp

button you will be presented with a list of Agents(Hosts) to select from or be required to

provide regexps for Agents and Metric selection - see next section

The TimeZone is that of the CA APM server This must be configured to ensure the metric data

is correctly mapped to UTC

Enter the connection information for the CA APM Host

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the CA

APM host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

32

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name

Give this destination a unique name This name will also be used as a prefix for the caapm CSV

files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

33

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from CA APM is based upon selecting the hosts and SWP will extract all

metrics for that host

Note If you had chosen to configure via regexp the list of hosts panel will be empty

You must provide a set of comma separated regexp pairs - one pair per line in the RegExp

panel eg

User

These are regexps for CA APM Agents and Metrics and are used directly in the queries eg

select from metric_data where agent= and metric=User

See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

34

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Note Sometimes depending on the responsiveness of the CA-APM system it may take a few

seconds for the list of hosts to appear Clicking in the panel often helps with refresh

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

35

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

36

Dynatrace Managed

Overview

This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api

Extraction Approach

Pre-requisites

The hostport and REST api connection token for your Dynatrace Managed system (see your

Admin andor Dynatrace API - Authentication)

Mediator creation and configuration

To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new origin

Press lsquo+rsquo on the SOURCES column to add a new source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

37

Now provide the connectivity information for your Dynatrace Managed system

Enter your Dynatrace Managed credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

Dynatrace Managed environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to produce IBM PI CSV files

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

38

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Select the desired Dynatrace Managed metric sources

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

39

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 8: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

8

The Mediation Pack provides the capability to extract performance data and metrics from the

following sources

bull SAP Hana DB

bull CA APM (Introscope)

bull MicroFocus NNMi

bull AWS Cloudwatch

bull Pivotal Cloud Foundry

bull Dynatrace Managed

bull AppDynamics

bull Oracle OEM

and deliver that data to IBM Operations Analytics Predictive Insights to perform analysis This

document will cover the installation and setup of Mediation Pack for each of the supported source-

data technologies

The figure shows an overall view of the mediation solution in context

Metrics are extracted by Collectors flow through to CSV Distributors which produce CSV files

The CSV files can then be made available to IBM PI for ingestion in the normal way Alternatively

any Collector can be routed to a PI REST Distributor which will deliver data to PI via its REST

interface

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

9

Prerequisite Information

Item Description

Name Mediation Pack Group 2

Predictive Insights supported

versions

v136 and higher Note IBM PI continues to support the CSV

format for ingestion and so earlier versions of PI ie 133 will

also work)

Predictive Insights Information IBM Operations Analytics (demo trial resources)

httpswwwibmcomus-enmarketplaceit-operations-

analytics

IBM Operations Analytics Community (docs videos

mediation packs) httpsdeveloperibmcomitoa

IBM Operations Analytics Predictive Insights

documentation

httpswwwibmcomsupportknowledgecenterSSJQQ3

Solution Name supported

versions

Min version Max Version

CA APM (Introscope) 9x 107

SAP Hana 10 10

AWS Cloudwatch EC2 REST

API 2016-11-

15

Cloudwatch

111

Same as Min

version

MF NNMi 1040x 1040x

Pivotal Cloud Foundry 233 233

Dynatrace Managed Dec 2018 Jun 2019

AppDynamics 41x 459

Oracle OEM 13 13

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

10

Solution Name Information IBM Operations Analytics Predictive Insights Mediation Pack

Group 2

Filename mediation_pack_group2-27-ibmpi-ltbuildNumbergtzip

Supported Operating Systems Linux

RHEL 6x 7x 3264bit

CentOS 6x7x 3264bit

Java Java18+ is required There is a version included in the release

package Note Mediators do NOT run on IBM JVM (the one

delivered with PI at time of writing)

Web Browsers A modern HTML5CSS3Javascript browser Versions of

Chrome Firefox or Safari since 2012 should work Previous

browser versions may not work well or at all

Note Some versions of Internet Explorer (ie 9 or earlier)

have exhibited issues Please consider a more modern

browser to ensure full functionality of the product

Ports Port 7457 (default but configurable) is required by

Mediators

Outbound ports per Mediator Type

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

11

SWP Mediators Overview

The metric data flow is generally from a collection component (a lsquoCollectorrsquo aka lsquoMediatorrsquo)

specific to the data source to an output component (a lsquoDistributorrsquo) Two distributors types are

support 1 CSV - writes metrics to standard IBM Operations Analytics Predictive Insights CSV files

2 REST - posts metric data (as JSON) to the IBM Operations Analytics Predictive Insights REST interface

Whilst each Mediator requires configuration specific to its source all the data flows through to a

corresponding Predictive Insights distributor The configurations approach for each of the

sources will be described in this document

The Mediators as part of the Mediation pack usually will be installed on the same host as

Predictive Insights This makes it easier to deliver the data from the Mediators to Predictive

Insights If the Mediation pack is installed on a different host then mechanisms must be put in

place to ensure that Predictive Insights can get at that CSV data eg shared disk drivesNFS

FTP-type jobs run on a schedule to retrieve files How to establish such mechanisms is beyond

the scope of this document In the case of REST distribution SWP does not have to be co-

located with Predictive Insights

Key steps to setting up Mediators

bull Installation

bull Configure one or more streams which extract the data from the supported sources

bull Configure the lsquoPredictive Insightsrsquo distributor stage for each stream choosing

between CSV and REST output

bull Start the desired metric extraction lsquotasksrsquo for each of the data sources This

should result in Predictive Insights CSV files being produced or JSON data

being posted to the IBM PI REST interface

bull Note Setting upconfiguring IBM PI REST interface is beyond the

scope of this document For details on this please refer to the

corresponding IBM PI documentation

bull In the case of CSV integration configure Predictive Insights for ingestion

using its lsquoMediation Toolrsquo in the normal way Sample Predictive Insights

lsquomodelrsquo files are provided as a starting point The lsquoMediation Toolrsquo is not

used for PI REST integration

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

12

Installing and Configuring

Zip file contents

The file mediation_pack_group2_v27-ibmpi- ltbuildgtzip contains the following

bull streamweaver-ibmpi-27ibmpiltbuildgt-1noarchrpm contains the main SWPMediators executable and associated collateral

bull docs containing the User Guides

o for lsquostreamweaverrsquo (this document)

o for Spl_AppD_Dyna_Ater_MedPack_v15_Revltrevisiongt

bull license containing license files

bull Spl_AppD_Dyna_Ater_MedPack_v15ltbuildgtrpm release with support for

o Dynatrace

o DCRum

o Aternity

o Splunk

o ExtraHop

o CA UIM

o MF APM

o NetIQ AppManager

o Note this package also contains support for AppDynamics Oracle OEM and CA APMIntroscope - these have been deprecated in 1x replaced by the Mediators in the main streamweaver 27 version

Install procedure

New Installation

The following section provides information for installing and configuring the Mediation Pack on

your IBM Operations Analytics Predictive Insights system

Normal lsquorpmrsquo conventions are followed

1 Verify Hardware amp Software Requirements

2 Verify user account privilege requirements

i The installation should be performed as the lsquorootrsquo user or user which can sudo ltgt

with root privileges This is required to create the Linux service

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

13

ii The installer will install as scadmindb2iadm (to match PI)

3 unzip mediation pack file

unzip mediation_pack_group2_v27-ibmpi-ltbuildgtzip

4 Install the bundled JVM

rpm -i --nodeps opstream-jre-18ibmpiltbuildgt-1x86_64rpm

5 Install SWP Mediators Locate the rpm file in the expanded zip directory

rpm -i streamweaver-ibmpi-27ibmpiltbuildgt-

1noarchrpm

After this you should see an indication of successful installation eg

SWP Mediators are installed by default to

optIBMstreamweaver-ibmpi

6 If you used root account exit and return to lsquoscadminrsquo user

Notes

Installing to an alternative location

You can install to a location other than optIBM as follows

sudo rpm -i --relocate optIBM=ltalternativepathgt streamweaver-

ibmpi-27ibmpiltbuildgt-1noarchrpm

Running as non-`scadmin` user

As part of the install if not already present in the system a `scadmin` account and lsquodb2iadm1` group are created for consistency with the default IBM PI installation and related file-ownerships

If you wish to run as a non-`scadmin` user

Change ownership of all the files from down eg

chown -R differentuserdifferentgroup optIBMstreamweaver-ibmpi

Edit the initd startup script

etcinitdstreamweaver-api

Locate and edit the line specifying the DAEMON_USER

[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=scadmin

Change the default `scadmin` to be your desired user eg

[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=differentuser

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

14

Co-existence of SWP-based and Accelerator-based mediators

If you are using the earlier Accelerator-based version along with this SWP-based version

note that they use the same default port setting of 7457 In this situation you should change the

default for SWP to something else You can do this by locating and editing the installed config

file in

optIBMstreamweaver-ibmpi confswpconf

and locate uncomment and change the line with port = 7457 to some other port

Settings for the Core Services services

Settings for the UI Service

ui

How the UI Services binds to a socket

connection

Hostname or IP Address to listen on The default

0000 causes

the REST API to listen on all interfaces

host = 0000

The port number to listen on

port = 7457

to some other free port

Also By default the UI listens on 7457 and the REST API listens on 7456 The UI assumes that the REST port is 1 less than the UI port Having made the above config change the following config entries are required and must be explicitly added to swpconf at the end of the file

httpport = ltrest-port-numbergt must be one less than UI port this defines the

port that REST API listens on lagomservicesswpallinone = httplocalhostltrest-port-

number server-side clients of REST service use this to locate it NOTE quotes are important here

Data directories

If you wish to relocate the various working directories (eg logs data) edit the swpconf file and modify the relevant entries in the ldquopathsrsquo section

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

15

Installation users

SWPMediators is installed by root as usergroup scadmindb2iadm1 to align the default PI usergroups

Starting SWP system

SWP contains Kafka Cassandra and the core They are all run under the lsquoswprsquo service

bull As lsquorootrsquo execute

service streamweaver-ibmpi start

In the usual way running lsquostatusrsquo on the service should give an indication that the main

SWP demon is operational

service streamweaver-ibmpi status

Once the SWP demon is running you use the SWP command line utility to monitor the

status of the child processes

Use lsquoswprsquo to observe the status of the system

bull optIBMstreamweaver-ibmpibinswp status

You should see a response similar to the following

In normal operation the five processes should be in Running state

You can also run the status command in a loop (like Linux `watch`) eg

bull optIBMstreamweaver-ibmpibinswp status -w 3

Stopping SWP system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

16

service streamweaver-ibmpi stop

Upgrade installation

service streamweaver-ibmpi stop

rpm -e streamweaver-ibmpi

Install new version as described above

Deployed directory contents

The main directories are created as part of the installation

optIBMstreamweaver-ibmpi

jre

The bundled JRE

bin

lsquoswpdrsquo all-in-one executable and lsquoswprsquo command line

content-packs

lsquoPI mediation lsquopa modelrsquo files and Pivotal CF lsquonozzlersquo components

lib

Library files (jars)

logs

Log files useful for monitoring and debugging system behavior

conf

Configuration files to control SWP

services

Cassandra Kafka and swpallinone (core SWP services)

Configuration files to control streamweaver

dataswp

Extracted metric data is stored under here

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

17

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

18

Accessing the Mediation system

Once installed you can interact with the Mediation system via web-browser By default the

Mediation system listens on port 7457 so for example point your web-browser to

httpltservergt7457

Weaver Pressing lsquoWEAVERrsquo brings you back to the above screen You select the lsquo+rsquo and a list

of source options will appear Any 3rd party tool listed can be selected to begin the configuration process In a similar fashion Destinations can also be configured by selecting the plus icon in the Destinations section

Dashboard Pressing lsquoDASHBOARDrsquo while cause latency measurements (the duration of last collection for each destination mediator) for each stream to be presented

Configuration Pressing lsquoDASHBOARDrsquo shows configuration information for the platform

including particularly log levels

System Logs Pressing lsquoSYSTEM LOGSrsquo presents the latest log entries (the full set of logs and be viewed in the relevant log directory which is described later in this document)

In the subsequent Mediator-specific sections we will give specific examples of

Mediator-gtDistributor Each example will specify a specific kind of distributor ie CSV or REST

You are free to use either distributor in each case

The initial step in each configuration case is to press lsquo+rsquo on the SOURCES and then you should

be presented with a list of available Mediator source types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

19

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

20

AppDynamics Mediator

Overview

Extraction Approach

Metric data is extracted using the AppDynamics REST API

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

AppD Port Defaults

REST 443

Mediator creation and configuration

To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your AppDynamics source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

21

Select Destination

Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single

file stream emerges from each Destination and we donrsquot generally want to mix data from

different Collector instances

Give this destination a unique name This name will also be used as a prefix for the

AppDynamics CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

22

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from AppDynamics is based upon selecting Applications Tiers and metric

paths SWP will extract all metrics for the supplied paths

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

23

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)

when data for the next time interval arrives

You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open

for the specified number of periods (as defined by the Maxium Time Window Per File) This is

useful when you want to wait for late data The trade-off is that the files are only made available

to PI after that lag period - thereby increasing end to end latency

For example if you had 5min files and set the max lag to 2 yoursquod have current file open and

two older files leading to up to 15 min latency

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

24

AWS Cloudwatch Mediator

Overview

This mediator extracts Metric data from AWS Cloudwatch system

Extraction Approach

Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference

Pre-requisites

You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users

The Endpoint regions are shown in the following table For the subsequent SWP configuration

you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account

has access to metric data

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

25

T

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

26

Mediator creation and configuration

To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the Source column to add a new origin

Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for AWS Cloudwatch account

Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least

one Region Multiple regions can be entered by separating each with a comma

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the AWS

CloudWatch environment)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

27

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

28

Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics

in those categories SWP will extract all metrics for those selected metric tables For IBM PI

only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are

selected

The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide

convenient resource groupings If you want to scope your metric extraction to particular tagged

instances specify those tags here as comma separated tagname=tagvalue pairs

For more details on AWS tagging see the AWS documentation

The optimum settings for the other configuration values depend on your AWS accountdata

availability etc The defaults should be sufficient to get you operational

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

29

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

30

CA APM (Introscope) Mediator

Overview

CA APM was formerly known as Wily Introscope

Metric extraction from CA APM is based upon configuring the system to extract data for specific

Hosts Metric data will be collected for all Agents and Processes on each selected host

Extraction Approach

Metric data is extracted using a JDBC interface

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

CA APM Port Defaults

JDBC 5001

Note

The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used

Mediator creation and configuration

To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

and then press lsquo+rsquo on the Sources column to add a new sources

Provide a name for the new source instance

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

31

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your CA APM DB

The Metadata Window Width and Latency control time window of metric data which is

extracted to determine the available monitored hosts for selection

Note It is also possible to configure selection to use regexp expressions which are used directly

in the JDBC queries sent to CA APM Depending on whether or not you check the regexp

button you will be presented with a list of Agents(Hosts) to select from or be required to

provide regexps for Agents and Metric selection - see next section

The TimeZone is that of the CA APM server This must be configured to ensure the metric data

is correctly mapped to UTC

Enter the connection information for the CA APM Host

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the CA

APM host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

32

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name

Give this destination a unique name This name will also be used as a prefix for the caapm CSV

files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

33

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from CA APM is based upon selecting the hosts and SWP will extract all

metrics for that host

Note If you had chosen to configure via regexp the list of hosts panel will be empty

You must provide a set of comma separated regexp pairs - one pair per line in the RegExp

panel eg

User

These are regexps for CA APM Agents and Metrics and are used directly in the queries eg

select from metric_data where agent= and metric=User

See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

34

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Note Sometimes depending on the responsiveness of the CA-APM system it may take a few

seconds for the list of hosts to appear Clicking in the panel often helps with refresh

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

35

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

36

Dynatrace Managed

Overview

This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api

Extraction Approach

Pre-requisites

The hostport and REST api connection token for your Dynatrace Managed system (see your

Admin andor Dynatrace API - Authentication)

Mediator creation and configuration

To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new origin

Press lsquo+rsquo on the SOURCES column to add a new source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

37

Now provide the connectivity information for your Dynatrace Managed system

Enter your Dynatrace Managed credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

Dynatrace Managed environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to produce IBM PI CSV files

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

38

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Select the desired Dynatrace Managed metric sources

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

39

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 9: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

9

Prerequisite Information

Item Description

Name Mediation Pack Group 2

Predictive Insights supported

versions

v136 and higher Note IBM PI continues to support the CSV

format for ingestion and so earlier versions of PI ie 133 will

also work)

Predictive Insights Information IBM Operations Analytics (demo trial resources)

httpswwwibmcomus-enmarketplaceit-operations-

analytics

IBM Operations Analytics Community (docs videos

mediation packs) httpsdeveloperibmcomitoa

IBM Operations Analytics Predictive Insights

documentation

httpswwwibmcomsupportknowledgecenterSSJQQ3

Solution Name supported

versions

Min version Max Version

CA APM (Introscope) 9x 107

SAP Hana 10 10

AWS Cloudwatch EC2 REST

API 2016-11-

15

Cloudwatch

111

Same as Min

version

MF NNMi 1040x 1040x

Pivotal Cloud Foundry 233 233

Dynatrace Managed Dec 2018 Jun 2019

AppDynamics 41x 459

Oracle OEM 13 13

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

10

Solution Name Information IBM Operations Analytics Predictive Insights Mediation Pack

Group 2

Filename mediation_pack_group2-27-ibmpi-ltbuildNumbergtzip

Supported Operating Systems Linux

RHEL 6x 7x 3264bit

CentOS 6x7x 3264bit

Java Java18+ is required There is a version included in the release

package Note Mediators do NOT run on IBM JVM (the one

delivered with PI at time of writing)

Web Browsers A modern HTML5CSS3Javascript browser Versions of

Chrome Firefox or Safari since 2012 should work Previous

browser versions may not work well or at all

Note Some versions of Internet Explorer (ie 9 or earlier)

have exhibited issues Please consider a more modern

browser to ensure full functionality of the product

Ports Port 7457 (default but configurable) is required by

Mediators

Outbound ports per Mediator Type

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

11

SWP Mediators Overview

The metric data flow is generally from a collection component (a lsquoCollectorrsquo aka lsquoMediatorrsquo)

specific to the data source to an output component (a lsquoDistributorrsquo) Two distributors types are

support 1 CSV - writes metrics to standard IBM Operations Analytics Predictive Insights CSV files

2 REST - posts metric data (as JSON) to the IBM Operations Analytics Predictive Insights REST interface

Whilst each Mediator requires configuration specific to its source all the data flows through to a

corresponding Predictive Insights distributor The configurations approach for each of the

sources will be described in this document

The Mediators as part of the Mediation pack usually will be installed on the same host as

Predictive Insights This makes it easier to deliver the data from the Mediators to Predictive

Insights If the Mediation pack is installed on a different host then mechanisms must be put in

place to ensure that Predictive Insights can get at that CSV data eg shared disk drivesNFS

FTP-type jobs run on a schedule to retrieve files How to establish such mechanisms is beyond

the scope of this document In the case of REST distribution SWP does not have to be co-

located with Predictive Insights

Key steps to setting up Mediators

bull Installation

bull Configure one or more streams which extract the data from the supported sources

bull Configure the lsquoPredictive Insightsrsquo distributor stage for each stream choosing

between CSV and REST output

bull Start the desired metric extraction lsquotasksrsquo for each of the data sources This

should result in Predictive Insights CSV files being produced or JSON data

being posted to the IBM PI REST interface

bull Note Setting upconfiguring IBM PI REST interface is beyond the

scope of this document For details on this please refer to the

corresponding IBM PI documentation

bull In the case of CSV integration configure Predictive Insights for ingestion

using its lsquoMediation Toolrsquo in the normal way Sample Predictive Insights

lsquomodelrsquo files are provided as a starting point The lsquoMediation Toolrsquo is not

used for PI REST integration

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

12

Installing and Configuring

Zip file contents

The file mediation_pack_group2_v27-ibmpi- ltbuildgtzip contains the following

bull streamweaver-ibmpi-27ibmpiltbuildgt-1noarchrpm contains the main SWPMediators executable and associated collateral

bull docs containing the User Guides

o for lsquostreamweaverrsquo (this document)

o for Spl_AppD_Dyna_Ater_MedPack_v15_Revltrevisiongt

bull license containing license files

bull Spl_AppD_Dyna_Ater_MedPack_v15ltbuildgtrpm release with support for

o Dynatrace

o DCRum

o Aternity

o Splunk

o ExtraHop

o CA UIM

o MF APM

o NetIQ AppManager

o Note this package also contains support for AppDynamics Oracle OEM and CA APMIntroscope - these have been deprecated in 1x replaced by the Mediators in the main streamweaver 27 version

Install procedure

New Installation

The following section provides information for installing and configuring the Mediation Pack on

your IBM Operations Analytics Predictive Insights system

Normal lsquorpmrsquo conventions are followed

1 Verify Hardware amp Software Requirements

2 Verify user account privilege requirements

i The installation should be performed as the lsquorootrsquo user or user which can sudo ltgt

with root privileges This is required to create the Linux service

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

13

ii The installer will install as scadmindb2iadm (to match PI)

3 unzip mediation pack file

unzip mediation_pack_group2_v27-ibmpi-ltbuildgtzip

4 Install the bundled JVM

rpm -i --nodeps opstream-jre-18ibmpiltbuildgt-1x86_64rpm

5 Install SWP Mediators Locate the rpm file in the expanded zip directory

rpm -i streamweaver-ibmpi-27ibmpiltbuildgt-

1noarchrpm

After this you should see an indication of successful installation eg

SWP Mediators are installed by default to

optIBMstreamweaver-ibmpi

6 If you used root account exit and return to lsquoscadminrsquo user

Notes

Installing to an alternative location

You can install to a location other than optIBM as follows

sudo rpm -i --relocate optIBM=ltalternativepathgt streamweaver-

ibmpi-27ibmpiltbuildgt-1noarchrpm

Running as non-`scadmin` user

As part of the install if not already present in the system a `scadmin` account and lsquodb2iadm1` group are created for consistency with the default IBM PI installation and related file-ownerships

If you wish to run as a non-`scadmin` user

Change ownership of all the files from down eg

chown -R differentuserdifferentgroup optIBMstreamweaver-ibmpi

Edit the initd startup script

etcinitdstreamweaver-api

Locate and edit the line specifying the DAEMON_USER

[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=scadmin

Change the default `scadmin` to be your desired user eg

[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=differentuser

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

14

Co-existence of SWP-based and Accelerator-based mediators

If you are using the earlier Accelerator-based version along with this SWP-based version

note that they use the same default port setting of 7457 In this situation you should change the

default for SWP to something else You can do this by locating and editing the installed config

file in

optIBMstreamweaver-ibmpi confswpconf

and locate uncomment and change the line with port = 7457 to some other port

Settings for the Core Services services

Settings for the UI Service

ui

How the UI Services binds to a socket

connection

Hostname or IP Address to listen on The default

0000 causes

the REST API to listen on all interfaces

host = 0000

The port number to listen on

port = 7457

to some other free port

Also By default the UI listens on 7457 and the REST API listens on 7456 The UI assumes that the REST port is 1 less than the UI port Having made the above config change the following config entries are required and must be explicitly added to swpconf at the end of the file

httpport = ltrest-port-numbergt must be one less than UI port this defines the

port that REST API listens on lagomservicesswpallinone = httplocalhostltrest-port-

number server-side clients of REST service use this to locate it NOTE quotes are important here

Data directories

If you wish to relocate the various working directories (eg logs data) edit the swpconf file and modify the relevant entries in the ldquopathsrsquo section

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

15

Installation users

SWPMediators is installed by root as usergroup scadmindb2iadm1 to align the default PI usergroups

Starting SWP system

SWP contains Kafka Cassandra and the core They are all run under the lsquoswprsquo service

bull As lsquorootrsquo execute

service streamweaver-ibmpi start

In the usual way running lsquostatusrsquo on the service should give an indication that the main

SWP demon is operational

service streamweaver-ibmpi status

Once the SWP demon is running you use the SWP command line utility to monitor the

status of the child processes

Use lsquoswprsquo to observe the status of the system

bull optIBMstreamweaver-ibmpibinswp status

You should see a response similar to the following

In normal operation the five processes should be in Running state

You can also run the status command in a loop (like Linux `watch`) eg

bull optIBMstreamweaver-ibmpibinswp status -w 3

Stopping SWP system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

16

service streamweaver-ibmpi stop

Upgrade installation

service streamweaver-ibmpi stop

rpm -e streamweaver-ibmpi

Install new version as described above

Deployed directory contents

The main directories are created as part of the installation

optIBMstreamweaver-ibmpi

jre

The bundled JRE

bin

lsquoswpdrsquo all-in-one executable and lsquoswprsquo command line

content-packs

lsquoPI mediation lsquopa modelrsquo files and Pivotal CF lsquonozzlersquo components

lib

Library files (jars)

logs

Log files useful for monitoring and debugging system behavior

conf

Configuration files to control SWP

services

Cassandra Kafka and swpallinone (core SWP services)

Configuration files to control streamweaver

dataswp

Extracted metric data is stored under here

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

17

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

18

Accessing the Mediation system

Once installed you can interact with the Mediation system via web-browser By default the

Mediation system listens on port 7457 so for example point your web-browser to

httpltservergt7457

Weaver Pressing lsquoWEAVERrsquo brings you back to the above screen You select the lsquo+rsquo and a list

of source options will appear Any 3rd party tool listed can be selected to begin the configuration process In a similar fashion Destinations can also be configured by selecting the plus icon in the Destinations section

Dashboard Pressing lsquoDASHBOARDrsquo while cause latency measurements (the duration of last collection for each destination mediator) for each stream to be presented

Configuration Pressing lsquoDASHBOARDrsquo shows configuration information for the platform

including particularly log levels

System Logs Pressing lsquoSYSTEM LOGSrsquo presents the latest log entries (the full set of logs and be viewed in the relevant log directory which is described later in this document)

In the subsequent Mediator-specific sections we will give specific examples of

Mediator-gtDistributor Each example will specify a specific kind of distributor ie CSV or REST

You are free to use either distributor in each case

The initial step in each configuration case is to press lsquo+rsquo on the SOURCES and then you should

be presented with a list of available Mediator source types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

19

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

20

AppDynamics Mediator

Overview

Extraction Approach

Metric data is extracted using the AppDynamics REST API

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

AppD Port Defaults

REST 443

Mediator creation and configuration

To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your AppDynamics source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

21

Select Destination

Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single

file stream emerges from each Destination and we donrsquot generally want to mix data from

different Collector instances

Give this destination a unique name This name will also be used as a prefix for the

AppDynamics CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

22

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from AppDynamics is based upon selecting Applications Tiers and metric

paths SWP will extract all metrics for the supplied paths

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

23

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)

when data for the next time interval arrives

You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open

for the specified number of periods (as defined by the Maxium Time Window Per File) This is

useful when you want to wait for late data The trade-off is that the files are only made available

to PI after that lag period - thereby increasing end to end latency

For example if you had 5min files and set the max lag to 2 yoursquod have current file open and

two older files leading to up to 15 min latency

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

24

AWS Cloudwatch Mediator

Overview

This mediator extracts Metric data from AWS Cloudwatch system

Extraction Approach

Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference

Pre-requisites

You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users

The Endpoint regions are shown in the following table For the subsequent SWP configuration

you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account

has access to metric data

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

25

T

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

26

Mediator creation and configuration

To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the Source column to add a new origin

Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for AWS Cloudwatch account

Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least

one Region Multiple regions can be entered by separating each with a comma

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the AWS

CloudWatch environment)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

27

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

28

Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics

in those categories SWP will extract all metrics for those selected metric tables For IBM PI

only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are

selected

The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide

convenient resource groupings If you want to scope your metric extraction to particular tagged

instances specify those tags here as comma separated tagname=tagvalue pairs

For more details on AWS tagging see the AWS documentation

The optimum settings for the other configuration values depend on your AWS accountdata

availability etc The defaults should be sufficient to get you operational

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

29

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

30

CA APM (Introscope) Mediator

Overview

CA APM was formerly known as Wily Introscope

Metric extraction from CA APM is based upon configuring the system to extract data for specific

Hosts Metric data will be collected for all Agents and Processes on each selected host

Extraction Approach

Metric data is extracted using a JDBC interface

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

CA APM Port Defaults

JDBC 5001

Note

The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used

Mediator creation and configuration

To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

and then press lsquo+rsquo on the Sources column to add a new sources

Provide a name for the new source instance

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

31

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your CA APM DB

The Metadata Window Width and Latency control time window of metric data which is

extracted to determine the available monitored hosts for selection

Note It is also possible to configure selection to use regexp expressions which are used directly

in the JDBC queries sent to CA APM Depending on whether or not you check the regexp

button you will be presented with a list of Agents(Hosts) to select from or be required to

provide regexps for Agents and Metric selection - see next section

The TimeZone is that of the CA APM server This must be configured to ensure the metric data

is correctly mapped to UTC

Enter the connection information for the CA APM Host

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the CA

APM host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

32

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name

Give this destination a unique name This name will also be used as a prefix for the caapm CSV

files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

33

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from CA APM is based upon selecting the hosts and SWP will extract all

metrics for that host

Note If you had chosen to configure via regexp the list of hosts panel will be empty

You must provide a set of comma separated regexp pairs - one pair per line in the RegExp

panel eg

User

These are regexps for CA APM Agents and Metrics and are used directly in the queries eg

select from metric_data where agent= and metric=User

See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

34

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Note Sometimes depending on the responsiveness of the CA-APM system it may take a few

seconds for the list of hosts to appear Clicking in the panel often helps with refresh

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

35

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

36

Dynatrace Managed

Overview

This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api

Extraction Approach

Pre-requisites

The hostport and REST api connection token for your Dynatrace Managed system (see your

Admin andor Dynatrace API - Authentication)

Mediator creation and configuration

To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new origin

Press lsquo+rsquo on the SOURCES column to add a new source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

37

Now provide the connectivity information for your Dynatrace Managed system

Enter your Dynatrace Managed credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

Dynatrace Managed environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to produce IBM PI CSV files

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

38

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Select the desired Dynatrace Managed metric sources

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

39

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 10: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

10

Solution Name Information IBM Operations Analytics Predictive Insights Mediation Pack

Group 2

Filename mediation_pack_group2-27-ibmpi-ltbuildNumbergtzip

Supported Operating Systems Linux

RHEL 6x 7x 3264bit

CentOS 6x7x 3264bit

Java Java18+ is required There is a version included in the release

package Note Mediators do NOT run on IBM JVM (the one

delivered with PI at time of writing)

Web Browsers A modern HTML5CSS3Javascript browser Versions of

Chrome Firefox or Safari since 2012 should work Previous

browser versions may not work well or at all

Note Some versions of Internet Explorer (ie 9 or earlier)

have exhibited issues Please consider a more modern

browser to ensure full functionality of the product

Ports Port 7457 (default but configurable) is required by

Mediators

Outbound ports per Mediator Type

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

11

SWP Mediators Overview

The metric data flow is generally from a collection component (a lsquoCollectorrsquo aka lsquoMediatorrsquo)

specific to the data source to an output component (a lsquoDistributorrsquo) Two distributors types are

support 1 CSV - writes metrics to standard IBM Operations Analytics Predictive Insights CSV files

2 REST - posts metric data (as JSON) to the IBM Operations Analytics Predictive Insights REST interface

Whilst each Mediator requires configuration specific to its source all the data flows through to a

corresponding Predictive Insights distributor The configurations approach for each of the

sources will be described in this document

The Mediators as part of the Mediation pack usually will be installed on the same host as

Predictive Insights This makes it easier to deliver the data from the Mediators to Predictive

Insights If the Mediation pack is installed on a different host then mechanisms must be put in

place to ensure that Predictive Insights can get at that CSV data eg shared disk drivesNFS

FTP-type jobs run on a schedule to retrieve files How to establish such mechanisms is beyond

the scope of this document In the case of REST distribution SWP does not have to be co-

located with Predictive Insights

Key steps to setting up Mediators

bull Installation

bull Configure one or more streams which extract the data from the supported sources

bull Configure the lsquoPredictive Insightsrsquo distributor stage for each stream choosing

between CSV and REST output

bull Start the desired metric extraction lsquotasksrsquo for each of the data sources This

should result in Predictive Insights CSV files being produced or JSON data

being posted to the IBM PI REST interface

bull Note Setting upconfiguring IBM PI REST interface is beyond the

scope of this document For details on this please refer to the

corresponding IBM PI documentation

bull In the case of CSV integration configure Predictive Insights for ingestion

using its lsquoMediation Toolrsquo in the normal way Sample Predictive Insights

lsquomodelrsquo files are provided as a starting point The lsquoMediation Toolrsquo is not

used for PI REST integration

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

12

Installing and Configuring

Zip file contents

The file mediation_pack_group2_v27-ibmpi- ltbuildgtzip contains the following

bull streamweaver-ibmpi-27ibmpiltbuildgt-1noarchrpm contains the main SWPMediators executable and associated collateral

bull docs containing the User Guides

o for lsquostreamweaverrsquo (this document)

o for Spl_AppD_Dyna_Ater_MedPack_v15_Revltrevisiongt

bull license containing license files

bull Spl_AppD_Dyna_Ater_MedPack_v15ltbuildgtrpm release with support for

o Dynatrace

o DCRum

o Aternity

o Splunk

o ExtraHop

o CA UIM

o MF APM

o NetIQ AppManager

o Note this package also contains support for AppDynamics Oracle OEM and CA APMIntroscope - these have been deprecated in 1x replaced by the Mediators in the main streamweaver 27 version

Install procedure

New Installation

The following section provides information for installing and configuring the Mediation Pack on

your IBM Operations Analytics Predictive Insights system

Normal lsquorpmrsquo conventions are followed

1 Verify Hardware amp Software Requirements

2 Verify user account privilege requirements

i The installation should be performed as the lsquorootrsquo user or user which can sudo ltgt

with root privileges This is required to create the Linux service

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

13

ii The installer will install as scadmindb2iadm (to match PI)

3 unzip mediation pack file

unzip mediation_pack_group2_v27-ibmpi-ltbuildgtzip

4 Install the bundled JVM

rpm -i --nodeps opstream-jre-18ibmpiltbuildgt-1x86_64rpm

5 Install SWP Mediators Locate the rpm file in the expanded zip directory

rpm -i streamweaver-ibmpi-27ibmpiltbuildgt-

1noarchrpm

After this you should see an indication of successful installation eg

SWP Mediators are installed by default to

optIBMstreamweaver-ibmpi

6 If you used root account exit and return to lsquoscadminrsquo user

Notes

Installing to an alternative location

You can install to a location other than optIBM as follows

sudo rpm -i --relocate optIBM=ltalternativepathgt streamweaver-

ibmpi-27ibmpiltbuildgt-1noarchrpm

Running as non-`scadmin` user

As part of the install if not already present in the system a `scadmin` account and lsquodb2iadm1` group are created for consistency with the default IBM PI installation and related file-ownerships

If you wish to run as a non-`scadmin` user

Change ownership of all the files from down eg

chown -R differentuserdifferentgroup optIBMstreamweaver-ibmpi

Edit the initd startup script

etcinitdstreamweaver-api

Locate and edit the line specifying the DAEMON_USER

[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=scadmin

Change the default `scadmin` to be your desired user eg

[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=differentuser

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

14

Co-existence of SWP-based and Accelerator-based mediators

If you are using the earlier Accelerator-based version along with this SWP-based version

note that they use the same default port setting of 7457 In this situation you should change the

default for SWP to something else You can do this by locating and editing the installed config

file in

optIBMstreamweaver-ibmpi confswpconf

and locate uncomment and change the line with port = 7457 to some other port

Settings for the Core Services services

Settings for the UI Service

ui

How the UI Services binds to a socket

connection

Hostname or IP Address to listen on The default

0000 causes

the REST API to listen on all interfaces

host = 0000

The port number to listen on

port = 7457

to some other free port

Also By default the UI listens on 7457 and the REST API listens on 7456 The UI assumes that the REST port is 1 less than the UI port Having made the above config change the following config entries are required and must be explicitly added to swpconf at the end of the file

httpport = ltrest-port-numbergt must be one less than UI port this defines the

port that REST API listens on lagomservicesswpallinone = httplocalhostltrest-port-

number server-side clients of REST service use this to locate it NOTE quotes are important here

Data directories

If you wish to relocate the various working directories (eg logs data) edit the swpconf file and modify the relevant entries in the ldquopathsrsquo section

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

15

Installation users

SWPMediators is installed by root as usergroup scadmindb2iadm1 to align the default PI usergroups

Starting SWP system

SWP contains Kafka Cassandra and the core They are all run under the lsquoswprsquo service

bull As lsquorootrsquo execute

service streamweaver-ibmpi start

In the usual way running lsquostatusrsquo on the service should give an indication that the main

SWP demon is operational

service streamweaver-ibmpi status

Once the SWP demon is running you use the SWP command line utility to monitor the

status of the child processes

Use lsquoswprsquo to observe the status of the system

bull optIBMstreamweaver-ibmpibinswp status

You should see a response similar to the following

In normal operation the five processes should be in Running state

You can also run the status command in a loop (like Linux `watch`) eg

bull optIBMstreamweaver-ibmpibinswp status -w 3

Stopping SWP system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

16

service streamweaver-ibmpi stop

Upgrade installation

service streamweaver-ibmpi stop

rpm -e streamweaver-ibmpi

Install new version as described above

Deployed directory contents

The main directories are created as part of the installation

optIBMstreamweaver-ibmpi

jre

The bundled JRE

bin

lsquoswpdrsquo all-in-one executable and lsquoswprsquo command line

content-packs

lsquoPI mediation lsquopa modelrsquo files and Pivotal CF lsquonozzlersquo components

lib

Library files (jars)

logs

Log files useful for monitoring and debugging system behavior

conf

Configuration files to control SWP

services

Cassandra Kafka and swpallinone (core SWP services)

Configuration files to control streamweaver

dataswp

Extracted metric data is stored under here

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

17

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

18

Accessing the Mediation system

Once installed you can interact with the Mediation system via web-browser By default the

Mediation system listens on port 7457 so for example point your web-browser to

httpltservergt7457

Weaver Pressing lsquoWEAVERrsquo brings you back to the above screen You select the lsquo+rsquo and a list

of source options will appear Any 3rd party tool listed can be selected to begin the configuration process In a similar fashion Destinations can also be configured by selecting the plus icon in the Destinations section

Dashboard Pressing lsquoDASHBOARDrsquo while cause latency measurements (the duration of last collection for each destination mediator) for each stream to be presented

Configuration Pressing lsquoDASHBOARDrsquo shows configuration information for the platform

including particularly log levels

System Logs Pressing lsquoSYSTEM LOGSrsquo presents the latest log entries (the full set of logs and be viewed in the relevant log directory which is described later in this document)

In the subsequent Mediator-specific sections we will give specific examples of

Mediator-gtDistributor Each example will specify a specific kind of distributor ie CSV or REST

You are free to use either distributor in each case

The initial step in each configuration case is to press lsquo+rsquo on the SOURCES and then you should

be presented with a list of available Mediator source types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

19

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

20

AppDynamics Mediator

Overview

Extraction Approach

Metric data is extracted using the AppDynamics REST API

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

AppD Port Defaults

REST 443

Mediator creation and configuration

To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your AppDynamics source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

21

Select Destination

Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single

file stream emerges from each Destination and we donrsquot generally want to mix data from

different Collector instances

Give this destination a unique name This name will also be used as a prefix for the

AppDynamics CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

22

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from AppDynamics is based upon selecting Applications Tiers and metric

paths SWP will extract all metrics for the supplied paths

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

23

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)

when data for the next time interval arrives

You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open

for the specified number of periods (as defined by the Maxium Time Window Per File) This is

useful when you want to wait for late data The trade-off is that the files are only made available

to PI after that lag period - thereby increasing end to end latency

For example if you had 5min files and set the max lag to 2 yoursquod have current file open and

two older files leading to up to 15 min latency

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

24

AWS Cloudwatch Mediator

Overview

This mediator extracts Metric data from AWS Cloudwatch system

Extraction Approach

Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference

Pre-requisites

You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users

The Endpoint regions are shown in the following table For the subsequent SWP configuration

you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account

has access to metric data

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

25

T

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

26

Mediator creation and configuration

To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the Source column to add a new origin

Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for AWS Cloudwatch account

Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least

one Region Multiple regions can be entered by separating each with a comma

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the AWS

CloudWatch environment)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

27

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

28

Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics

in those categories SWP will extract all metrics for those selected metric tables For IBM PI

only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are

selected

The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide

convenient resource groupings If you want to scope your metric extraction to particular tagged

instances specify those tags here as comma separated tagname=tagvalue pairs

For more details on AWS tagging see the AWS documentation

The optimum settings for the other configuration values depend on your AWS accountdata

availability etc The defaults should be sufficient to get you operational

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

29

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

30

CA APM (Introscope) Mediator

Overview

CA APM was formerly known as Wily Introscope

Metric extraction from CA APM is based upon configuring the system to extract data for specific

Hosts Metric data will be collected for all Agents and Processes on each selected host

Extraction Approach

Metric data is extracted using a JDBC interface

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

CA APM Port Defaults

JDBC 5001

Note

The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used

Mediator creation and configuration

To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

and then press lsquo+rsquo on the Sources column to add a new sources

Provide a name for the new source instance

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

31

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your CA APM DB

The Metadata Window Width and Latency control time window of metric data which is

extracted to determine the available monitored hosts for selection

Note It is also possible to configure selection to use regexp expressions which are used directly

in the JDBC queries sent to CA APM Depending on whether or not you check the regexp

button you will be presented with a list of Agents(Hosts) to select from or be required to

provide regexps for Agents and Metric selection - see next section

The TimeZone is that of the CA APM server This must be configured to ensure the metric data

is correctly mapped to UTC

Enter the connection information for the CA APM Host

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the CA

APM host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

32

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name

Give this destination a unique name This name will also be used as a prefix for the caapm CSV

files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

33

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from CA APM is based upon selecting the hosts and SWP will extract all

metrics for that host

Note If you had chosen to configure via regexp the list of hosts panel will be empty

You must provide a set of comma separated regexp pairs - one pair per line in the RegExp

panel eg

User

These are regexps for CA APM Agents and Metrics and are used directly in the queries eg

select from metric_data where agent= and metric=User

See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

34

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Note Sometimes depending on the responsiveness of the CA-APM system it may take a few

seconds for the list of hosts to appear Clicking in the panel often helps with refresh

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

35

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

36

Dynatrace Managed

Overview

This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api

Extraction Approach

Pre-requisites

The hostport and REST api connection token for your Dynatrace Managed system (see your

Admin andor Dynatrace API - Authentication)

Mediator creation and configuration

To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new origin

Press lsquo+rsquo on the SOURCES column to add a new source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

37

Now provide the connectivity information for your Dynatrace Managed system

Enter your Dynatrace Managed credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

Dynatrace Managed environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to produce IBM PI CSV files

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

38

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Select the desired Dynatrace Managed metric sources

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

39

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 11: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

11

SWP Mediators Overview

The metric data flow is generally from a collection component (a lsquoCollectorrsquo aka lsquoMediatorrsquo)

specific to the data source to an output component (a lsquoDistributorrsquo) Two distributors types are

support 1 CSV - writes metrics to standard IBM Operations Analytics Predictive Insights CSV files

2 REST - posts metric data (as JSON) to the IBM Operations Analytics Predictive Insights REST interface

Whilst each Mediator requires configuration specific to its source all the data flows through to a

corresponding Predictive Insights distributor The configurations approach for each of the

sources will be described in this document

The Mediators as part of the Mediation pack usually will be installed on the same host as

Predictive Insights This makes it easier to deliver the data from the Mediators to Predictive

Insights If the Mediation pack is installed on a different host then mechanisms must be put in

place to ensure that Predictive Insights can get at that CSV data eg shared disk drivesNFS

FTP-type jobs run on a schedule to retrieve files How to establish such mechanisms is beyond

the scope of this document In the case of REST distribution SWP does not have to be co-

located with Predictive Insights

Key steps to setting up Mediators

bull Installation

bull Configure one or more streams which extract the data from the supported sources

bull Configure the lsquoPredictive Insightsrsquo distributor stage for each stream choosing

between CSV and REST output

bull Start the desired metric extraction lsquotasksrsquo for each of the data sources This

should result in Predictive Insights CSV files being produced or JSON data

being posted to the IBM PI REST interface

bull Note Setting upconfiguring IBM PI REST interface is beyond the

scope of this document For details on this please refer to the

corresponding IBM PI documentation

bull In the case of CSV integration configure Predictive Insights for ingestion

using its lsquoMediation Toolrsquo in the normal way Sample Predictive Insights

lsquomodelrsquo files are provided as a starting point The lsquoMediation Toolrsquo is not

used for PI REST integration

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

12

Installing and Configuring

Zip file contents

The file mediation_pack_group2_v27-ibmpi- ltbuildgtzip contains the following

bull streamweaver-ibmpi-27ibmpiltbuildgt-1noarchrpm contains the main SWPMediators executable and associated collateral

bull docs containing the User Guides

o for lsquostreamweaverrsquo (this document)

o for Spl_AppD_Dyna_Ater_MedPack_v15_Revltrevisiongt

bull license containing license files

bull Spl_AppD_Dyna_Ater_MedPack_v15ltbuildgtrpm release with support for

o Dynatrace

o DCRum

o Aternity

o Splunk

o ExtraHop

o CA UIM

o MF APM

o NetIQ AppManager

o Note this package also contains support for AppDynamics Oracle OEM and CA APMIntroscope - these have been deprecated in 1x replaced by the Mediators in the main streamweaver 27 version

Install procedure

New Installation

The following section provides information for installing and configuring the Mediation Pack on

your IBM Operations Analytics Predictive Insights system

Normal lsquorpmrsquo conventions are followed

1 Verify Hardware amp Software Requirements

2 Verify user account privilege requirements

i The installation should be performed as the lsquorootrsquo user or user which can sudo ltgt

with root privileges This is required to create the Linux service

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

13

ii The installer will install as scadmindb2iadm (to match PI)

3 unzip mediation pack file

unzip mediation_pack_group2_v27-ibmpi-ltbuildgtzip

4 Install the bundled JVM

rpm -i --nodeps opstream-jre-18ibmpiltbuildgt-1x86_64rpm

5 Install SWP Mediators Locate the rpm file in the expanded zip directory

rpm -i streamweaver-ibmpi-27ibmpiltbuildgt-

1noarchrpm

After this you should see an indication of successful installation eg

SWP Mediators are installed by default to

optIBMstreamweaver-ibmpi

6 If you used root account exit and return to lsquoscadminrsquo user

Notes

Installing to an alternative location

You can install to a location other than optIBM as follows

sudo rpm -i --relocate optIBM=ltalternativepathgt streamweaver-

ibmpi-27ibmpiltbuildgt-1noarchrpm

Running as non-`scadmin` user

As part of the install if not already present in the system a `scadmin` account and lsquodb2iadm1` group are created for consistency with the default IBM PI installation and related file-ownerships

If you wish to run as a non-`scadmin` user

Change ownership of all the files from down eg

chown -R differentuserdifferentgroup optIBMstreamweaver-ibmpi

Edit the initd startup script

etcinitdstreamweaver-api

Locate and edit the line specifying the DAEMON_USER

[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=scadmin

Change the default `scadmin` to be your desired user eg

[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=differentuser

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

14

Co-existence of SWP-based and Accelerator-based mediators

If you are using the earlier Accelerator-based version along with this SWP-based version

note that they use the same default port setting of 7457 In this situation you should change the

default for SWP to something else You can do this by locating and editing the installed config

file in

optIBMstreamweaver-ibmpi confswpconf

and locate uncomment and change the line with port = 7457 to some other port

Settings for the Core Services services

Settings for the UI Service

ui

How the UI Services binds to a socket

connection

Hostname or IP Address to listen on The default

0000 causes

the REST API to listen on all interfaces

host = 0000

The port number to listen on

port = 7457

to some other free port

Also By default the UI listens on 7457 and the REST API listens on 7456 The UI assumes that the REST port is 1 less than the UI port Having made the above config change the following config entries are required and must be explicitly added to swpconf at the end of the file

httpport = ltrest-port-numbergt must be one less than UI port this defines the

port that REST API listens on lagomservicesswpallinone = httplocalhostltrest-port-

number server-side clients of REST service use this to locate it NOTE quotes are important here

Data directories

If you wish to relocate the various working directories (eg logs data) edit the swpconf file and modify the relevant entries in the ldquopathsrsquo section

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

15

Installation users

SWPMediators is installed by root as usergroup scadmindb2iadm1 to align the default PI usergroups

Starting SWP system

SWP contains Kafka Cassandra and the core They are all run under the lsquoswprsquo service

bull As lsquorootrsquo execute

service streamweaver-ibmpi start

In the usual way running lsquostatusrsquo on the service should give an indication that the main

SWP demon is operational

service streamweaver-ibmpi status

Once the SWP demon is running you use the SWP command line utility to monitor the

status of the child processes

Use lsquoswprsquo to observe the status of the system

bull optIBMstreamweaver-ibmpibinswp status

You should see a response similar to the following

In normal operation the five processes should be in Running state

You can also run the status command in a loop (like Linux `watch`) eg

bull optIBMstreamweaver-ibmpibinswp status -w 3

Stopping SWP system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

16

service streamweaver-ibmpi stop

Upgrade installation

service streamweaver-ibmpi stop

rpm -e streamweaver-ibmpi

Install new version as described above

Deployed directory contents

The main directories are created as part of the installation

optIBMstreamweaver-ibmpi

jre

The bundled JRE

bin

lsquoswpdrsquo all-in-one executable and lsquoswprsquo command line

content-packs

lsquoPI mediation lsquopa modelrsquo files and Pivotal CF lsquonozzlersquo components

lib

Library files (jars)

logs

Log files useful for monitoring and debugging system behavior

conf

Configuration files to control SWP

services

Cassandra Kafka and swpallinone (core SWP services)

Configuration files to control streamweaver

dataswp

Extracted metric data is stored under here

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

17

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

18

Accessing the Mediation system

Once installed you can interact with the Mediation system via web-browser By default the

Mediation system listens on port 7457 so for example point your web-browser to

httpltservergt7457

Weaver Pressing lsquoWEAVERrsquo brings you back to the above screen You select the lsquo+rsquo and a list

of source options will appear Any 3rd party tool listed can be selected to begin the configuration process In a similar fashion Destinations can also be configured by selecting the plus icon in the Destinations section

Dashboard Pressing lsquoDASHBOARDrsquo while cause latency measurements (the duration of last collection for each destination mediator) for each stream to be presented

Configuration Pressing lsquoDASHBOARDrsquo shows configuration information for the platform

including particularly log levels

System Logs Pressing lsquoSYSTEM LOGSrsquo presents the latest log entries (the full set of logs and be viewed in the relevant log directory which is described later in this document)

In the subsequent Mediator-specific sections we will give specific examples of

Mediator-gtDistributor Each example will specify a specific kind of distributor ie CSV or REST

You are free to use either distributor in each case

The initial step in each configuration case is to press lsquo+rsquo on the SOURCES and then you should

be presented with a list of available Mediator source types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

19

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

20

AppDynamics Mediator

Overview

Extraction Approach

Metric data is extracted using the AppDynamics REST API

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

AppD Port Defaults

REST 443

Mediator creation and configuration

To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your AppDynamics source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

21

Select Destination

Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single

file stream emerges from each Destination and we donrsquot generally want to mix data from

different Collector instances

Give this destination a unique name This name will also be used as a prefix for the

AppDynamics CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

22

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from AppDynamics is based upon selecting Applications Tiers and metric

paths SWP will extract all metrics for the supplied paths

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

23

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)

when data for the next time interval arrives

You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open

for the specified number of periods (as defined by the Maxium Time Window Per File) This is

useful when you want to wait for late data The trade-off is that the files are only made available

to PI after that lag period - thereby increasing end to end latency

For example if you had 5min files and set the max lag to 2 yoursquod have current file open and

two older files leading to up to 15 min latency

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

24

AWS Cloudwatch Mediator

Overview

This mediator extracts Metric data from AWS Cloudwatch system

Extraction Approach

Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference

Pre-requisites

You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users

The Endpoint regions are shown in the following table For the subsequent SWP configuration

you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account

has access to metric data

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

25

T

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

26

Mediator creation and configuration

To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the Source column to add a new origin

Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for AWS Cloudwatch account

Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least

one Region Multiple regions can be entered by separating each with a comma

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the AWS

CloudWatch environment)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

27

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

28

Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics

in those categories SWP will extract all metrics for those selected metric tables For IBM PI

only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are

selected

The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide

convenient resource groupings If you want to scope your metric extraction to particular tagged

instances specify those tags here as comma separated tagname=tagvalue pairs

For more details on AWS tagging see the AWS documentation

The optimum settings for the other configuration values depend on your AWS accountdata

availability etc The defaults should be sufficient to get you operational

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

29

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

30

CA APM (Introscope) Mediator

Overview

CA APM was formerly known as Wily Introscope

Metric extraction from CA APM is based upon configuring the system to extract data for specific

Hosts Metric data will be collected for all Agents and Processes on each selected host

Extraction Approach

Metric data is extracted using a JDBC interface

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

CA APM Port Defaults

JDBC 5001

Note

The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used

Mediator creation and configuration

To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

and then press lsquo+rsquo on the Sources column to add a new sources

Provide a name for the new source instance

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

31

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your CA APM DB

The Metadata Window Width and Latency control time window of metric data which is

extracted to determine the available monitored hosts for selection

Note It is also possible to configure selection to use regexp expressions which are used directly

in the JDBC queries sent to CA APM Depending on whether or not you check the regexp

button you will be presented with a list of Agents(Hosts) to select from or be required to

provide regexps for Agents and Metric selection - see next section

The TimeZone is that of the CA APM server This must be configured to ensure the metric data

is correctly mapped to UTC

Enter the connection information for the CA APM Host

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the CA

APM host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

32

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name

Give this destination a unique name This name will also be used as a prefix for the caapm CSV

files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

33

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from CA APM is based upon selecting the hosts and SWP will extract all

metrics for that host

Note If you had chosen to configure via regexp the list of hosts panel will be empty

You must provide a set of comma separated regexp pairs - one pair per line in the RegExp

panel eg

User

These are regexps for CA APM Agents and Metrics and are used directly in the queries eg

select from metric_data where agent= and metric=User

See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

34

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Note Sometimes depending on the responsiveness of the CA-APM system it may take a few

seconds for the list of hosts to appear Clicking in the panel often helps with refresh

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

35

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

36

Dynatrace Managed

Overview

This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api

Extraction Approach

Pre-requisites

The hostport and REST api connection token for your Dynatrace Managed system (see your

Admin andor Dynatrace API - Authentication)

Mediator creation and configuration

To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new origin

Press lsquo+rsquo on the SOURCES column to add a new source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

37

Now provide the connectivity information for your Dynatrace Managed system

Enter your Dynatrace Managed credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

Dynatrace Managed environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to produce IBM PI CSV files

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

38

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Select the desired Dynatrace Managed metric sources

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

39

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 12: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

12

Installing and Configuring

Zip file contents

The file mediation_pack_group2_v27-ibmpi- ltbuildgtzip contains the following

bull streamweaver-ibmpi-27ibmpiltbuildgt-1noarchrpm contains the main SWPMediators executable and associated collateral

bull docs containing the User Guides

o for lsquostreamweaverrsquo (this document)

o for Spl_AppD_Dyna_Ater_MedPack_v15_Revltrevisiongt

bull license containing license files

bull Spl_AppD_Dyna_Ater_MedPack_v15ltbuildgtrpm release with support for

o Dynatrace

o DCRum

o Aternity

o Splunk

o ExtraHop

o CA UIM

o MF APM

o NetIQ AppManager

o Note this package also contains support for AppDynamics Oracle OEM and CA APMIntroscope - these have been deprecated in 1x replaced by the Mediators in the main streamweaver 27 version

Install procedure

New Installation

The following section provides information for installing and configuring the Mediation Pack on

your IBM Operations Analytics Predictive Insights system

Normal lsquorpmrsquo conventions are followed

1 Verify Hardware amp Software Requirements

2 Verify user account privilege requirements

i The installation should be performed as the lsquorootrsquo user or user which can sudo ltgt

with root privileges This is required to create the Linux service

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

13

ii The installer will install as scadmindb2iadm (to match PI)

3 unzip mediation pack file

unzip mediation_pack_group2_v27-ibmpi-ltbuildgtzip

4 Install the bundled JVM

rpm -i --nodeps opstream-jre-18ibmpiltbuildgt-1x86_64rpm

5 Install SWP Mediators Locate the rpm file in the expanded zip directory

rpm -i streamweaver-ibmpi-27ibmpiltbuildgt-

1noarchrpm

After this you should see an indication of successful installation eg

SWP Mediators are installed by default to

optIBMstreamweaver-ibmpi

6 If you used root account exit and return to lsquoscadminrsquo user

Notes

Installing to an alternative location

You can install to a location other than optIBM as follows

sudo rpm -i --relocate optIBM=ltalternativepathgt streamweaver-

ibmpi-27ibmpiltbuildgt-1noarchrpm

Running as non-`scadmin` user

As part of the install if not already present in the system a `scadmin` account and lsquodb2iadm1` group are created for consistency with the default IBM PI installation and related file-ownerships

If you wish to run as a non-`scadmin` user

Change ownership of all the files from down eg

chown -R differentuserdifferentgroup optIBMstreamweaver-ibmpi

Edit the initd startup script

etcinitdstreamweaver-api

Locate and edit the line specifying the DAEMON_USER

[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=scadmin

Change the default `scadmin` to be your desired user eg

[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=differentuser

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

14

Co-existence of SWP-based and Accelerator-based mediators

If you are using the earlier Accelerator-based version along with this SWP-based version

note that they use the same default port setting of 7457 In this situation you should change the

default for SWP to something else You can do this by locating and editing the installed config

file in

optIBMstreamweaver-ibmpi confswpconf

and locate uncomment and change the line with port = 7457 to some other port

Settings for the Core Services services

Settings for the UI Service

ui

How the UI Services binds to a socket

connection

Hostname or IP Address to listen on The default

0000 causes

the REST API to listen on all interfaces

host = 0000

The port number to listen on

port = 7457

to some other free port

Also By default the UI listens on 7457 and the REST API listens on 7456 The UI assumes that the REST port is 1 less than the UI port Having made the above config change the following config entries are required and must be explicitly added to swpconf at the end of the file

httpport = ltrest-port-numbergt must be one less than UI port this defines the

port that REST API listens on lagomservicesswpallinone = httplocalhostltrest-port-

number server-side clients of REST service use this to locate it NOTE quotes are important here

Data directories

If you wish to relocate the various working directories (eg logs data) edit the swpconf file and modify the relevant entries in the ldquopathsrsquo section

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

15

Installation users

SWPMediators is installed by root as usergroup scadmindb2iadm1 to align the default PI usergroups

Starting SWP system

SWP contains Kafka Cassandra and the core They are all run under the lsquoswprsquo service

bull As lsquorootrsquo execute

service streamweaver-ibmpi start

In the usual way running lsquostatusrsquo on the service should give an indication that the main

SWP demon is operational

service streamweaver-ibmpi status

Once the SWP demon is running you use the SWP command line utility to monitor the

status of the child processes

Use lsquoswprsquo to observe the status of the system

bull optIBMstreamweaver-ibmpibinswp status

You should see a response similar to the following

In normal operation the five processes should be in Running state

You can also run the status command in a loop (like Linux `watch`) eg

bull optIBMstreamweaver-ibmpibinswp status -w 3

Stopping SWP system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

16

service streamweaver-ibmpi stop

Upgrade installation

service streamweaver-ibmpi stop

rpm -e streamweaver-ibmpi

Install new version as described above

Deployed directory contents

The main directories are created as part of the installation

optIBMstreamweaver-ibmpi

jre

The bundled JRE

bin

lsquoswpdrsquo all-in-one executable and lsquoswprsquo command line

content-packs

lsquoPI mediation lsquopa modelrsquo files and Pivotal CF lsquonozzlersquo components

lib

Library files (jars)

logs

Log files useful for monitoring and debugging system behavior

conf

Configuration files to control SWP

services

Cassandra Kafka and swpallinone (core SWP services)

Configuration files to control streamweaver

dataswp

Extracted metric data is stored under here

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

17

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

18

Accessing the Mediation system

Once installed you can interact with the Mediation system via web-browser By default the

Mediation system listens on port 7457 so for example point your web-browser to

httpltservergt7457

Weaver Pressing lsquoWEAVERrsquo brings you back to the above screen You select the lsquo+rsquo and a list

of source options will appear Any 3rd party tool listed can be selected to begin the configuration process In a similar fashion Destinations can also be configured by selecting the plus icon in the Destinations section

Dashboard Pressing lsquoDASHBOARDrsquo while cause latency measurements (the duration of last collection for each destination mediator) for each stream to be presented

Configuration Pressing lsquoDASHBOARDrsquo shows configuration information for the platform

including particularly log levels

System Logs Pressing lsquoSYSTEM LOGSrsquo presents the latest log entries (the full set of logs and be viewed in the relevant log directory which is described later in this document)

In the subsequent Mediator-specific sections we will give specific examples of

Mediator-gtDistributor Each example will specify a specific kind of distributor ie CSV or REST

You are free to use either distributor in each case

The initial step in each configuration case is to press lsquo+rsquo on the SOURCES and then you should

be presented with a list of available Mediator source types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

19

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

20

AppDynamics Mediator

Overview

Extraction Approach

Metric data is extracted using the AppDynamics REST API

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

AppD Port Defaults

REST 443

Mediator creation and configuration

To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your AppDynamics source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

21

Select Destination

Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single

file stream emerges from each Destination and we donrsquot generally want to mix data from

different Collector instances

Give this destination a unique name This name will also be used as a prefix for the

AppDynamics CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

22

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from AppDynamics is based upon selecting Applications Tiers and metric

paths SWP will extract all metrics for the supplied paths

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

23

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)

when data for the next time interval arrives

You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open

for the specified number of periods (as defined by the Maxium Time Window Per File) This is

useful when you want to wait for late data The trade-off is that the files are only made available

to PI after that lag period - thereby increasing end to end latency

For example if you had 5min files and set the max lag to 2 yoursquod have current file open and

two older files leading to up to 15 min latency

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

24

AWS Cloudwatch Mediator

Overview

This mediator extracts Metric data from AWS Cloudwatch system

Extraction Approach

Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference

Pre-requisites

You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users

The Endpoint regions are shown in the following table For the subsequent SWP configuration

you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account

has access to metric data

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

25

T

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

26

Mediator creation and configuration

To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the Source column to add a new origin

Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for AWS Cloudwatch account

Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least

one Region Multiple regions can be entered by separating each with a comma

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the AWS

CloudWatch environment)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

27

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

28

Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics

in those categories SWP will extract all metrics for those selected metric tables For IBM PI

only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are

selected

The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide

convenient resource groupings If you want to scope your metric extraction to particular tagged

instances specify those tags here as comma separated tagname=tagvalue pairs

For more details on AWS tagging see the AWS documentation

The optimum settings for the other configuration values depend on your AWS accountdata

availability etc The defaults should be sufficient to get you operational

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

29

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

30

CA APM (Introscope) Mediator

Overview

CA APM was formerly known as Wily Introscope

Metric extraction from CA APM is based upon configuring the system to extract data for specific

Hosts Metric data will be collected for all Agents and Processes on each selected host

Extraction Approach

Metric data is extracted using a JDBC interface

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

CA APM Port Defaults

JDBC 5001

Note

The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used

Mediator creation and configuration

To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

and then press lsquo+rsquo on the Sources column to add a new sources

Provide a name for the new source instance

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

31

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your CA APM DB

The Metadata Window Width and Latency control time window of metric data which is

extracted to determine the available monitored hosts for selection

Note It is also possible to configure selection to use regexp expressions which are used directly

in the JDBC queries sent to CA APM Depending on whether or not you check the regexp

button you will be presented with a list of Agents(Hosts) to select from or be required to

provide regexps for Agents and Metric selection - see next section

The TimeZone is that of the CA APM server This must be configured to ensure the metric data

is correctly mapped to UTC

Enter the connection information for the CA APM Host

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the CA

APM host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

32

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name

Give this destination a unique name This name will also be used as a prefix for the caapm CSV

files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

33

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from CA APM is based upon selecting the hosts and SWP will extract all

metrics for that host

Note If you had chosen to configure via regexp the list of hosts panel will be empty

You must provide a set of comma separated regexp pairs - one pair per line in the RegExp

panel eg

User

These are regexps for CA APM Agents and Metrics and are used directly in the queries eg

select from metric_data where agent= and metric=User

See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

34

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Note Sometimes depending on the responsiveness of the CA-APM system it may take a few

seconds for the list of hosts to appear Clicking in the panel often helps with refresh

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

35

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

36

Dynatrace Managed

Overview

This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api

Extraction Approach

Pre-requisites

The hostport and REST api connection token for your Dynatrace Managed system (see your

Admin andor Dynatrace API - Authentication)

Mediator creation and configuration

To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new origin

Press lsquo+rsquo on the SOURCES column to add a new source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

37

Now provide the connectivity information for your Dynatrace Managed system

Enter your Dynatrace Managed credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

Dynatrace Managed environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to produce IBM PI CSV files

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

38

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Select the desired Dynatrace Managed metric sources

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

39

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 13: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

13

ii The installer will install as scadmindb2iadm (to match PI)

3 unzip mediation pack file

unzip mediation_pack_group2_v27-ibmpi-ltbuildgtzip

4 Install the bundled JVM

rpm -i --nodeps opstream-jre-18ibmpiltbuildgt-1x86_64rpm

5 Install SWP Mediators Locate the rpm file in the expanded zip directory

rpm -i streamweaver-ibmpi-27ibmpiltbuildgt-

1noarchrpm

After this you should see an indication of successful installation eg

SWP Mediators are installed by default to

optIBMstreamweaver-ibmpi

6 If you used root account exit and return to lsquoscadminrsquo user

Notes

Installing to an alternative location

You can install to a location other than optIBM as follows

sudo rpm -i --relocate optIBM=ltalternativepathgt streamweaver-

ibmpi-27ibmpiltbuildgt-1noarchrpm

Running as non-`scadmin` user

As part of the install if not already present in the system a `scadmin` account and lsquodb2iadm1` group are created for consistency with the default IBM PI installation and related file-ownerships

If you wish to run as a non-`scadmin` user

Change ownership of all the files from down eg

chown -R differentuserdifferentgroup optIBMstreamweaver-ibmpi

Edit the initd startup script

etcinitdstreamweaver-api

Locate and edit the line specifying the DAEMON_USER

[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=scadmin

Change the default `scadmin` to be your desired user eg

[ -z ldquo$DAEMON_USER-rdquo ] ampamp DAEMON_USER=differentuser

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

14

Co-existence of SWP-based and Accelerator-based mediators

If you are using the earlier Accelerator-based version along with this SWP-based version

note that they use the same default port setting of 7457 In this situation you should change the

default for SWP to something else You can do this by locating and editing the installed config

file in

optIBMstreamweaver-ibmpi confswpconf

and locate uncomment and change the line with port = 7457 to some other port

Settings for the Core Services services

Settings for the UI Service

ui

How the UI Services binds to a socket

connection

Hostname or IP Address to listen on The default

0000 causes

the REST API to listen on all interfaces

host = 0000

The port number to listen on

port = 7457

to some other free port

Also By default the UI listens on 7457 and the REST API listens on 7456 The UI assumes that the REST port is 1 less than the UI port Having made the above config change the following config entries are required and must be explicitly added to swpconf at the end of the file

httpport = ltrest-port-numbergt must be one less than UI port this defines the

port that REST API listens on lagomservicesswpallinone = httplocalhostltrest-port-

number server-side clients of REST service use this to locate it NOTE quotes are important here

Data directories

If you wish to relocate the various working directories (eg logs data) edit the swpconf file and modify the relevant entries in the ldquopathsrsquo section

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

15

Installation users

SWPMediators is installed by root as usergroup scadmindb2iadm1 to align the default PI usergroups

Starting SWP system

SWP contains Kafka Cassandra and the core They are all run under the lsquoswprsquo service

bull As lsquorootrsquo execute

service streamweaver-ibmpi start

In the usual way running lsquostatusrsquo on the service should give an indication that the main

SWP demon is operational

service streamweaver-ibmpi status

Once the SWP demon is running you use the SWP command line utility to monitor the

status of the child processes

Use lsquoswprsquo to observe the status of the system

bull optIBMstreamweaver-ibmpibinswp status

You should see a response similar to the following

In normal operation the five processes should be in Running state

You can also run the status command in a loop (like Linux `watch`) eg

bull optIBMstreamweaver-ibmpibinswp status -w 3

Stopping SWP system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

16

service streamweaver-ibmpi stop

Upgrade installation

service streamweaver-ibmpi stop

rpm -e streamweaver-ibmpi

Install new version as described above

Deployed directory contents

The main directories are created as part of the installation

optIBMstreamweaver-ibmpi

jre

The bundled JRE

bin

lsquoswpdrsquo all-in-one executable and lsquoswprsquo command line

content-packs

lsquoPI mediation lsquopa modelrsquo files and Pivotal CF lsquonozzlersquo components

lib

Library files (jars)

logs

Log files useful for monitoring and debugging system behavior

conf

Configuration files to control SWP

services

Cassandra Kafka and swpallinone (core SWP services)

Configuration files to control streamweaver

dataswp

Extracted metric data is stored under here

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

17

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

18

Accessing the Mediation system

Once installed you can interact with the Mediation system via web-browser By default the

Mediation system listens on port 7457 so for example point your web-browser to

httpltservergt7457

Weaver Pressing lsquoWEAVERrsquo brings you back to the above screen You select the lsquo+rsquo and a list

of source options will appear Any 3rd party tool listed can be selected to begin the configuration process In a similar fashion Destinations can also be configured by selecting the plus icon in the Destinations section

Dashboard Pressing lsquoDASHBOARDrsquo while cause latency measurements (the duration of last collection for each destination mediator) for each stream to be presented

Configuration Pressing lsquoDASHBOARDrsquo shows configuration information for the platform

including particularly log levels

System Logs Pressing lsquoSYSTEM LOGSrsquo presents the latest log entries (the full set of logs and be viewed in the relevant log directory which is described later in this document)

In the subsequent Mediator-specific sections we will give specific examples of

Mediator-gtDistributor Each example will specify a specific kind of distributor ie CSV or REST

You are free to use either distributor in each case

The initial step in each configuration case is to press lsquo+rsquo on the SOURCES and then you should

be presented with a list of available Mediator source types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

19

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

20

AppDynamics Mediator

Overview

Extraction Approach

Metric data is extracted using the AppDynamics REST API

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

AppD Port Defaults

REST 443

Mediator creation and configuration

To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your AppDynamics source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

21

Select Destination

Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single

file stream emerges from each Destination and we donrsquot generally want to mix data from

different Collector instances

Give this destination a unique name This name will also be used as a prefix for the

AppDynamics CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

22

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from AppDynamics is based upon selecting Applications Tiers and metric

paths SWP will extract all metrics for the supplied paths

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

23

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)

when data for the next time interval arrives

You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open

for the specified number of periods (as defined by the Maxium Time Window Per File) This is

useful when you want to wait for late data The trade-off is that the files are only made available

to PI after that lag period - thereby increasing end to end latency

For example if you had 5min files and set the max lag to 2 yoursquod have current file open and

two older files leading to up to 15 min latency

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

24

AWS Cloudwatch Mediator

Overview

This mediator extracts Metric data from AWS Cloudwatch system

Extraction Approach

Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference

Pre-requisites

You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users

The Endpoint regions are shown in the following table For the subsequent SWP configuration

you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account

has access to metric data

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

25

T

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

26

Mediator creation and configuration

To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the Source column to add a new origin

Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for AWS Cloudwatch account

Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least

one Region Multiple regions can be entered by separating each with a comma

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the AWS

CloudWatch environment)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

27

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

28

Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics

in those categories SWP will extract all metrics for those selected metric tables For IBM PI

only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are

selected

The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide

convenient resource groupings If you want to scope your metric extraction to particular tagged

instances specify those tags here as comma separated tagname=tagvalue pairs

For more details on AWS tagging see the AWS documentation

The optimum settings for the other configuration values depend on your AWS accountdata

availability etc The defaults should be sufficient to get you operational

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

29

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

30

CA APM (Introscope) Mediator

Overview

CA APM was formerly known as Wily Introscope

Metric extraction from CA APM is based upon configuring the system to extract data for specific

Hosts Metric data will be collected for all Agents and Processes on each selected host

Extraction Approach

Metric data is extracted using a JDBC interface

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

CA APM Port Defaults

JDBC 5001

Note

The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used

Mediator creation and configuration

To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

and then press lsquo+rsquo on the Sources column to add a new sources

Provide a name for the new source instance

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

31

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your CA APM DB

The Metadata Window Width and Latency control time window of metric data which is

extracted to determine the available monitored hosts for selection

Note It is also possible to configure selection to use regexp expressions which are used directly

in the JDBC queries sent to CA APM Depending on whether or not you check the regexp

button you will be presented with a list of Agents(Hosts) to select from or be required to

provide regexps for Agents and Metric selection - see next section

The TimeZone is that of the CA APM server This must be configured to ensure the metric data

is correctly mapped to UTC

Enter the connection information for the CA APM Host

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the CA

APM host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

32

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name

Give this destination a unique name This name will also be used as a prefix for the caapm CSV

files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

33

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from CA APM is based upon selecting the hosts and SWP will extract all

metrics for that host

Note If you had chosen to configure via regexp the list of hosts panel will be empty

You must provide a set of comma separated regexp pairs - one pair per line in the RegExp

panel eg

User

These are regexps for CA APM Agents and Metrics and are used directly in the queries eg

select from metric_data where agent= and metric=User

See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

34

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Note Sometimes depending on the responsiveness of the CA-APM system it may take a few

seconds for the list of hosts to appear Clicking in the panel often helps with refresh

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

35

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

36

Dynatrace Managed

Overview

This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api

Extraction Approach

Pre-requisites

The hostport and REST api connection token for your Dynatrace Managed system (see your

Admin andor Dynatrace API - Authentication)

Mediator creation and configuration

To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new origin

Press lsquo+rsquo on the SOURCES column to add a new source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

37

Now provide the connectivity information for your Dynatrace Managed system

Enter your Dynatrace Managed credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

Dynatrace Managed environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to produce IBM PI CSV files

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

38

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Select the desired Dynatrace Managed metric sources

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

39

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 14: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

14

Co-existence of SWP-based and Accelerator-based mediators

If you are using the earlier Accelerator-based version along with this SWP-based version

note that they use the same default port setting of 7457 In this situation you should change the

default for SWP to something else You can do this by locating and editing the installed config

file in

optIBMstreamweaver-ibmpi confswpconf

and locate uncomment and change the line with port = 7457 to some other port

Settings for the Core Services services

Settings for the UI Service

ui

How the UI Services binds to a socket

connection

Hostname or IP Address to listen on The default

0000 causes

the REST API to listen on all interfaces

host = 0000

The port number to listen on

port = 7457

to some other free port

Also By default the UI listens on 7457 and the REST API listens on 7456 The UI assumes that the REST port is 1 less than the UI port Having made the above config change the following config entries are required and must be explicitly added to swpconf at the end of the file

httpport = ltrest-port-numbergt must be one less than UI port this defines the

port that REST API listens on lagomservicesswpallinone = httplocalhostltrest-port-

number server-side clients of REST service use this to locate it NOTE quotes are important here

Data directories

If you wish to relocate the various working directories (eg logs data) edit the swpconf file and modify the relevant entries in the ldquopathsrsquo section

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

15

Installation users

SWPMediators is installed by root as usergroup scadmindb2iadm1 to align the default PI usergroups

Starting SWP system

SWP contains Kafka Cassandra and the core They are all run under the lsquoswprsquo service

bull As lsquorootrsquo execute

service streamweaver-ibmpi start

In the usual way running lsquostatusrsquo on the service should give an indication that the main

SWP demon is operational

service streamweaver-ibmpi status

Once the SWP demon is running you use the SWP command line utility to monitor the

status of the child processes

Use lsquoswprsquo to observe the status of the system

bull optIBMstreamweaver-ibmpibinswp status

You should see a response similar to the following

In normal operation the five processes should be in Running state

You can also run the status command in a loop (like Linux `watch`) eg

bull optIBMstreamweaver-ibmpibinswp status -w 3

Stopping SWP system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

16

service streamweaver-ibmpi stop

Upgrade installation

service streamweaver-ibmpi stop

rpm -e streamweaver-ibmpi

Install new version as described above

Deployed directory contents

The main directories are created as part of the installation

optIBMstreamweaver-ibmpi

jre

The bundled JRE

bin

lsquoswpdrsquo all-in-one executable and lsquoswprsquo command line

content-packs

lsquoPI mediation lsquopa modelrsquo files and Pivotal CF lsquonozzlersquo components

lib

Library files (jars)

logs

Log files useful for monitoring and debugging system behavior

conf

Configuration files to control SWP

services

Cassandra Kafka and swpallinone (core SWP services)

Configuration files to control streamweaver

dataswp

Extracted metric data is stored under here

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

17

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

18

Accessing the Mediation system

Once installed you can interact with the Mediation system via web-browser By default the

Mediation system listens on port 7457 so for example point your web-browser to

httpltservergt7457

Weaver Pressing lsquoWEAVERrsquo brings you back to the above screen You select the lsquo+rsquo and a list

of source options will appear Any 3rd party tool listed can be selected to begin the configuration process In a similar fashion Destinations can also be configured by selecting the plus icon in the Destinations section

Dashboard Pressing lsquoDASHBOARDrsquo while cause latency measurements (the duration of last collection for each destination mediator) for each stream to be presented

Configuration Pressing lsquoDASHBOARDrsquo shows configuration information for the platform

including particularly log levels

System Logs Pressing lsquoSYSTEM LOGSrsquo presents the latest log entries (the full set of logs and be viewed in the relevant log directory which is described later in this document)

In the subsequent Mediator-specific sections we will give specific examples of

Mediator-gtDistributor Each example will specify a specific kind of distributor ie CSV or REST

You are free to use either distributor in each case

The initial step in each configuration case is to press lsquo+rsquo on the SOURCES and then you should

be presented with a list of available Mediator source types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

19

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

20

AppDynamics Mediator

Overview

Extraction Approach

Metric data is extracted using the AppDynamics REST API

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

AppD Port Defaults

REST 443

Mediator creation and configuration

To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your AppDynamics source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

21

Select Destination

Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single

file stream emerges from each Destination and we donrsquot generally want to mix data from

different Collector instances

Give this destination a unique name This name will also be used as a prefix for the

AppDynamics CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

22

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from AppDynamics is based upon selecting Applications Tiers and metric

paths SWP will extract all metrics for the supplied paths

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

23

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)

when data for the next time interval arrives

You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open

for the specified number of periods (as defined by the Maxium Time Window Per File) This is

useful when you want to wait for late data The trade-off is that the files are only made available

to PI after that lag period - thereby increasing end to end latency

For example if you had 5min files and set the max lag to 2 yoursquod have current file open and

two older files leading to up to 15 min latency

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

24

AWS Cloudwatch Mediator

Overview

This mediator extracts Metric data from AWS Cloudwatch system

Extraction Approach

Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference

Pre-requisites

You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users

The Endpoint regions are shown in the following table For the subsequent SWP configuration

you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account

has access to metric data

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

25

T

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

26

Mediator creation and configuration

To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the Source column to add a new origin

Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for AWS Cloudwatch account

Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least

one Region Multiple regions can be entered by separating each with a comma

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the AWS

CloudWatch environment)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

27

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

28

Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics

in those categories SWP will extract all metrics for those selected metric tables For IBM PI

only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are

selected

The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide

convenient resource groupings If you want to scope your metric extraction to particular tagged

instances specify those tags here as comma separated tagname=tagvalue pairs

For more details on AWS tagging see the AWS documentation

The optimum settings for the other configuration values depend on your AWS accountdata

availability etc The defaults should be sufficient to get you operational

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

29

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

30

CA APM (Introscope) Mediator

Overview

CA APM was formerly known as Wily Introscope

Metric extraction from CA APM is based upon configuring the system to extract data for specific

Hosts Metric data will be collected for all Agents and Processes on each selected host

Extraction Approach

Metric data is extracted using a JDBC interface

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

CA APM Port Defaults

JDBC 5001

Note

The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used

Mediator creation and configuration

To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

and then press lsquo+rsquo on the Sources column to add a new sources

Provide a name for the new source instance

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

31

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your CA APM DB

The Metadata Window Width and Latency control time window of metric data which is

extracted to determine the available monitored hosts for selection

Note It is also possible to configure selection to use regexp expressions which are used directly

in the JDBC queries sent to CA APM Depending on whether or not you check the regexp

button you will be presented with a list of Agents(Hosts) to select from or be required to

provide regexps for Agents and Metric selection - see next section

The TimeZone is that of the CA APM server This must be configured to ensure the metric data

is correctly mapped to UTC

Enter the connection information for the CA APM Host

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the CA

APM host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

32

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name

Give this destination a unique name This name will also be used as a prefix for the caapm CSV

files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

33

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from CA APM is based upon selecting the hosts and SWP will extract all

metrics for that host

Note If you had chosen to configure via regexp the list of hosts panel will be empty

You must provide a set of comma separated regexp pairs - one pair per line in the RegExp

panel eg

User

These are regexps for CA APM Agents and Metrics and are used directly in the queries eg

select from metric_data where agent= and metric=User

See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

34

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Note Sometimes depending on the responsiveness of the CA-APM system it may take a few

seconds for the list of hosts to appear Clicking in the panel often helps with refresh

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

35

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

36

Dynatrace Managed

Overview

This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api

Extraction Approach

Pre-requisites

The hostport and REST api connection token for your Dynatrace Managed system (see your

Admin andor Dynatrace API - Authentication)

Mediator creation and configuration

To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new origin

Press lsquo+rsquo on the SOURCES column to add a new source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

37

Now provide the connectivity information for your Dynatrace Managed system

Enter your Dynatrace Managed credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

Dynatrace Managed environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to produce IBM PI CSV files

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

38

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Select the desired Dynatrace Managed metric sources

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

39

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 15: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

15

Installation users

SWPMediators is installed by root as usergroup scadmindb2iadm1 to align the default PI usergroups

Starting SWP system

SWP contains Kafka Cassandra and the core They are all run under the lsquoswprsquo service

bull As lsquorootrsquo execute

service streamweaver-ibmpi start

In the usual way running lsquostatusrsquo on the service should give an indication that the main

SWP demon is operational

service streamweaver-ibmpi status

Once the SWP demon is running you use the SWP command line utility to monitor the

status of the child processes

Use lsquoswprsquo to observe the status of the system

bull optIBMstreamweaver-ibmpibinswp status

You should see a response similar to the following

In normal operation the five processes should be in Running state

You can also run the status command in a loop (like Linux `watch`) eg

bull optIBMstreamweaver-ibmpibinswp status -w 3

Stopping SWP system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

16

service streamweaver-ibmpi stop

Upgrade installation

service streamweaver-ibmpi stop

rpm -e streamweaver-ibmpi

Install new version as described above

Deployed directory contents

The main directories are created as part of the installation

optIBMstreamweaver-ibmpi

jre

The bundled JRE

bin

lsquoswpdrsquo all-in-one executable and lsquoswprsquo command line

content-packs

lsquoPI mediation lsquopa modelrsquo files and Pivotal CF lsquonozzlersquo components

lib

Library files (jars)

logs

Log files useful for monitoring and debugging system behavior

conf

Configuration files to control SWP

services

Cassandra Kafka and swpallinone (core SWP services)

Configuration files to control streamweaver

dataswp

Extracted metric data is stored under here

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

17

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

18

Accessing the Mediation system

Once installed you can interact with the Mediation system via web-browser By default the

Mediation system listens on port 7457 so for example point your web-browser to

httpltservergt7457

Weaver Pressing lsquoWEAVERrsquo brings you back to the above screen You select the lsquo+rsquo and a list

of source options will appear Any 3rd party tool listed can be selected to begin the configuration process In a similar fashion Destinations can also be configured by selecting the plus icon in the Destinations section

Dashboard Pressing lsquoDASHBOARDrsquo while cause latency measurements (the duration of last collection for each destination mediator) for each stream to be presented

Configuration Pressing lsquoDASHBOARDrsquo shows configuration information for the platform

including particularly log levels

System Logs Pressing lsquoSYSTEM LOGSrsquo presents the latest log entries (the full set of logs and be viewed in the relevant log directory which is described later in this document)

In the subsequent Mediator-specific sections we will give specific examples of

Mediator-gtDistributor Each example will specify a specific kind of distributor ie CSV or REST

You are free to use either distributor in each case

The initial step in each configuration case is to press lsquo+rsquo on the SOURCES and then you should

be presented with a list of available Mediator source types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

19

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

20

AppDynamics Mediator

Overview

Extraction Approach

Metric data is extracted using the AppDynamics REST API

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

AppD Port Defaults

REST 443

Mediator creation and configuration

To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your AppDynamics source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

21

Select Destination

Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single

file stream emerges from each Destination and we donrsquot generally want to mix data from

different Collector instances

Give this destination a unique name This name will also be used as a prefix for the

AppDynamics CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

22

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from AppDynamics is based upon selecting Applications Tiers and metric

paths SWP will extract all metrics for the supplied paths

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

23

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)

when data for the next time interval arrives

You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open

for the specified number of periods (as defined by the Maxium Time Window Per File) This is

useful when you want to wait for late data The trade-off is that the files are only made available

to PI after that lag period - thereby increasing end to end latency

For example if you had 5min files and set the max lag to 2 yoursquod have current file open and

two older files leading to up to 15 min latency

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

24

AWS Cloudwatch Mediator

Overview

This mediator extracts Metric data from AWS Cloudwatch system

Extraction Approach

Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference

Pre-requisites

You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users

The Endpoint regions are shown in the following table For the subsequent SWP configuration

you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account

has access to metric data

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

25

T

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

26

Mediator creation and configuration

To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the Source column to add a new origin

Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for AWS Cloudwatch account

Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least

one Region Multiple regions can be entered by separating each with a comma

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the AWS

CloudWatch environment)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

27

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

28

Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics

in those categories SWP will extract all metrics for those selected metric tables For IBM PI

only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are

selected

The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide

convenient resource groupings If you want to scope your metric extraction to particular tagged

instances specify those tags here as comma separated tagname=tagvalue pairs

For more details on AWS tagging see the AWS documentation

The optimum settings for the other configuration values depend on your AWS accountdata

availability etc The defaults should be sufficient to get you operational

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

29

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

30

CA APM (Introscope) Mediator

Overview

CA APM was formerly known as Wily Introscope

Metric extraction from CA APM is based upon configuring the system to extract data for specific

Hosts Metric data will be collected for all Agents and Processes on each selected host

Extraction Approach

Metric data is extracted using a JDBC interface

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

CA APM Port Defaults

JDBC 5001

Note

The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used

Mediator creation and configuration

To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

and then press lsquo+rsquo on the Sources column to add a new sources

Provide a name for the new source instance

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

31

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your CA APM DB

The Metadata Window Width and Latency control time window of metric data which is

extracted to determine the available monitored hosts for selection

Note It is also possible to configure selection to use regexp expressions which are used directly

in the JDBC queries sent to CA APM Depending on whether or not you check the regexp

button you will be presented with a list of Agents(Hosts) to select from or be required to

provide regexps for Agents and Metric selection - see next section

The TimeZone is that of the CA APM server This must be configured to ensure the metric data

is correctly mapped to UTC

Enter the connection information for the CA APM Host

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the CA

APM host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

32

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name

Give this destination a unique name This name will also be used as a prefix for the caapm CSV

files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

33

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from CA APM is based upon selecting the hosts and SWP will extract all

metrics for that host

Note If you had chosen to configure via regexp the list of hosts panel will be empty

You must provide a set of comma separated regexp pairs - one pair per line in the RegExp

panel eg

User

These are regexps for CA APM Agents and Metrics and are used directly in the queries eg

select from metric_data where agent= and metric=User

See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

34

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Note Sometimes depending on the responsiveness of the CA-APM system it may take a few

seconds for the list of hosts to appear Clicking in the panel often helps with refresh

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

35

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

36

Dynatrace Managed

Overview

This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api

Extraction Approach

Pre-requisites

The hostport and REST api connection token for your Dynatrace Managed system (see your

Admin andor Dynatrace API - Authentication)

Mediator creation and configuration

To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new origin

Press lsquo+rsquo on the SOURCES column to add a new source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

37

Now provide the connectivity information for your Dynatrace Managed system

Enter your Dynatrace Managed credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

Dynatrace Managed environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to produce IBM PI CSV files

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

38

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Select the desired Dynatrace Managed metric sources

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

39

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 16: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

16

service streamweaver-ibmpi stop

Upgrade installation

service streamweaver-ibmpi stop

rpm -e streamweaver-ibmpi

Install new version as described above

Deployed directory contents

The main directories are created as part of the installation

optIBMstreamweaver-ibmpi

jre

The bundled JRE

bin

lsquoswpdrsquo all-in-one executable and lsquoswprsquo command line

content-packs

lsquoPI mediation lsquopa modelrsquo files and Pivotal CF lsquonozzlersquo components

lib

Library files (jars)

logs

Log files useful for monitoring and debugging system behavior

conf

Configuration files to control SWP

services

Cassandra Kafka and swpallinone (core SWP services)

Configuration files to control streamweaver

dataswp

Extracted metric data is stored under here

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

17

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

18

Accessing the Mediation system

Once installed you can interact with the Mediation system via web-browser By default the

Mediation system listens on port 7457 so for example point your web-browser to

httpltservergt7457

Weaver Pressing lsquoWEAVERrsquo brings you back to the above screen You select the lsquo+rsquo and a list

of source options will appear Any 3rd party tool listed can be selected to begin the configuration process In a similar fashion Destinations can also be configured by selecting the plus icon in the Destinations section

Dashboard Pressing lsquoDASHBOARDrsquo while cause latency measurements (the duration of last collection for each destination mediator) for each stream to be presented

Configuration Pressing lsquoDASHBOARDrsquo shows configuration information for the platform

including particularly log levels

System Logs Pressing lsquoSYSTEM LOGSrsquo presents the latest log entries (the full set of logs and be viewed in the relevant log directory which is described later in this document)

In the subsequent Mediator-specific sections we will give specific examples of

Mediator-gtDistributor Each example will specify a specific kind of distributor ie CSV or REST

You are free to use either distributor in each case

The initial step in each configuration case is to press lsquo+rsquo on the SOURCES and then you should

be presented with a list of available Mediator source types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

19

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

20

AppDynamics Mediator

Overview

Extraction Approach

Metric data is extracted using the AppDynamics REST API

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

AppD Port Defaults

REST 443

Mediator creation and configuration

To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your AppDynamics source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

21

Select Destination

Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single

file stream emerges from each Destination and we donrsquot generally want to mix data from

different Collector instances

Give this destination a unique name This name will also be used as a prefix for the

AppDynamics CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

22

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from AppDynamics is based upon selecting Applications Tiers and metric

paths SWP will extract all metrics for the supplied paths

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

23

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)

when data for the next time interval arrives

You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open

for the specified number of periods (as defined by the Maxium Time Window Per File) This is

useful when you want to wait for late data The trade-off is that the files are only made available

to PI after that lag period - thereby increasing end to end latency

For example if you had 5min files and set the max lag to 2 yoursquod have current file open and

two older files leading to up to 15 min latency

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

24

AWS Cloudwatch Mediator

Overview

This mediator extracts Metric data from AWS Cloudwatch system

Extraction Approach

Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference

Pre-requisites

You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users

The Endpoint regions are shown in the following table For the subsequent SWP configuration

you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account

has access to metric data

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

25

T

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

26

Mediator creation and configuration

To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the Source column to add a new origin

Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for AWS Cloudwatch account

Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least

one Region Multiple regions can be entered by separating each with a comma

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the AWS

CloudWatch environment)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

27

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

28

Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics

in those categories SWP will extract all metrics for those selected metric tables For IBM PI

only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are

selected

The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide

convenient resource groupings If you want to scope your metric extraction to particular tagged

instances specify those tags here as comma separated tagname=tagvalue pairs

For more details on AWS tagging see the AWS documentation

The optimum settings for the other configuration values depend on your AWS accountdata

availability etc The defaults should be sufficient to get you operational

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

29

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

30

CA APM (Introscope) Mediator

Overview

CA APM was formerly known as Wily Introscope

Metric extraction from CA APM is based upon configuring the system to extract data for specific

Hosts Metric data will be collected for all Agents and Processes on each selected host

Extraction Approach

Metric data is extracted using a JDBC interface

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

CA APM Port Defaults

JDBC 5001

Note

The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used

Mediator creation and configuration

To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

and then press lsquo+rsquo on the Sources column to add a new sources

Provide a name for the new source instance

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

31

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your CA APM DB

The Metadata Window Width and Latency control time window of metric data which is

extracted to determine the available monitored hosts for selection

Note It is also possible to configure selection to use regexp expressions which are used directly

in the JDBC queries sent to CA APM Depending on whether or not you check the regexp

button you will be presented with a list of Agents(Hosts) to select from or be required to

provide regexps for Agents and Metric selection - see next section

The TimeZone is that of the CA APM server This must be configured to ensure the metric data

is correctly mapped to UTC

Enter the connection information for the CA APM Host

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the CA

APM host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

32

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name

Give this destination a unique name This name will also be used as a prefix for the caapm CSV

files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

33

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from CA APM is based upon selecting the hosts and SWP will extract all

metrics for that host

Note If you had chosen to configure via regexp the list of hosts panel will be empty

You must provide a set of comma separated regexp pairs - one pair per line in the RegExp

panel eg

User

These are regexps for CA APM Agents and Metrics and are used directly in the queries eg

select from metric_data where agent= and metric=User

See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

34

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Note Sometimes depending on the responsiveness of the CA-APM system it may take a few

seconds for the list of hosts to appear Clicking in the panel often helps with refresh

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

35

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

36

Dynatrace Managed

Overview

This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api

Extraction Approach

Pre-requisites

The hostport and REST api connection token for your Dynatrace Managed system (see your

Admin andor Dynatrace API - Authentication)

Mediator creation and configuration

To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new origin

Press lsquo+rsquo on the SOURCES column to add a new source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

37

Now provide the connectivity information for your Dynatrace Managed system

Enter your Dynatrace Managed credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

Dynatrace Managed environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to produce IBM PI CSV files

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

38

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Select the desired Dynatrace Managed metric sources

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

39

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 17: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

17

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

18

Accessing the Mediation system

Once installed you can interact with the Mediation system via web-browser By default the

Mediation system listens on port 7457 so for example point your web-browser to

httpltservergt7457

Weaver Pressing lsquoWEAVERrsquo brings you back to the above screen You select the lsquo+rsquo and a list

of source options will appear Any 3rd party tool listed can be selected to begin the configuration process In a similar fashion Destinations can also be configured by selecting the plus icon in the Destinations section

Dashboard Pressing lsquoDASHBOARDrsquo while cause latency measurements (the duration of last collection for each destination mediator) for each stream to be presented

Configuration Pressing lsquoDASHBOARDrsquo shows configuration information for the platform

including particularly log levels

System Logs Pressing lsquoSYSTEM LOGSrsquo presents the latest log entries (the full set of logs and be viewed in the relevant log directory which is described later in this document)

In the subsequent Mediator-specific sections we will give specific examples of

Mediator-gtDistributor Each example will specify a specific kind of distributor ie CSV or REST

You are free to use either distributor in each case

The initial step in each configuration case is to press lsquo+rsquo on the SOURCES and then you should

be presented with a list of available Mediator source types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

19

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

20

AppDynamics Mediator

Overview

Extraction Approach

Metric data is extracted using the AppDynamics REST API

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

AppD Port Defaults

REST 443

Mediator creation and configuration

To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your AppDynamics source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

21

Select Destination

Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single

file stream emerges from each Destination and we donrsquot generally want to mix data from

different Collector instances

Give this destination a unique name This name will also be used as a prefix for the

AppDynamics CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

22

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from AppDynamics is based upon selecting Applications Tiers and metric

paths SWP will extract all metrics for the supplied paths

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

23

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)

when data for the next time interval arrives

You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open

for the specified number of periods (as defined by the Maxium Time Window Per File) This is

useful when you want to wait for late data The trade-off is that the files are only made available

to PI after that lag period - thereby increasing end to end latency

For example if you had 5min files and set the max lag to 2 yoursquod have current file open and

two older files leading to up to 15 min latency

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

24

AWS Cloudwatch Mediator

Overview

This mediator extracts Metric data from AWS Cloudwatch system

Extraction Approach

Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference

Pre-requisites

You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users

The Endpoint regions are shown in the following table For the subsequent SWP configuration

you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account

has access to metric data

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

25

T

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

26

Mediator creation and configuration

To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the Source column to add a new origin

Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for AWS Cloudwatch account

Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least

one Region Multiple regions can be entered by separating each with a comma

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the AWS

CloudWatch environment)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

27

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

28

Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics

in those categories SWP will extract all metrics for those selected metric tables For IBM PI

only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are

selected

The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide

convenient resource groupings If you want to scope your metric extraction to particular tagged

instances specify those tags here as comma separated tagname=tagvalue pairs

For more details on AWS tagging see the AWS documentation

The optimum settings for the other configuration values depend on your AWS accountdata

availability etc The defaults should be sufficient to get you operational

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

29

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

30

CA APM (Introscope) Mediator

Overview

CA APM was formerly known as Wily Introscope

Metric extraction from CA APM is based upon configuring the system to extract data for specific

Hosts Metric data will be collected for all Agents and Processes on each selected host

Extraction Approach

Metric data is extracted using a JDBC interface

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

CA APM Port Defaults

JDBC 5001

Note

The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used

Mediator creation and configuration

To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

and then press lsquo+rsquo on the Sources column to add a new sources

Provide a name for the new source instance

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

31

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your CA APM DB

The Metadata Window Width and Latency control time window of metric data which is

extracted to determine the available monitored hosts for selection

Note It is also possible to configure selection to use regexp expressions which are used directly

in the JDBC queries sent to CA APM Depending on whether or not you check the regexp

button you will be presented with a list of Agents(Hosts) to select from or be required to

provide regexps for Agents and Metric selection - see next section

The TimeZone is that of the CA APM server This must be configured to ensure the metric data

is correctly mapped to UTC

Enter the connection information for the CA APM Host

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the CA

APM host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

32

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name

Give this destination a unique name This name will also be used as a prefix for the caapm CSV

files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

33

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from CA APM is based upon selecting the hosts and SWP will extract all

metrics for that host

Note If you had chosen to configure via regexp the list of hosts panel will be empty

You must provide a set of comma separated regexp pairs - one pair per line in the RegExp

panel eg

User

These are regexps for CA APM Agents and Metrics and are used directly in the queries eg

select from metric_data where agent= and metric=User

See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

34

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Note Sometimes depending on the responsiveness of the CA-APM system it may take a few

seconds for the list of hosts to appear Clicking in the panel often helps with refresh

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

35

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

36

Dynatrace Managed

Overview

This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api

Extraction Approach

Pre-requisites

The hostport and REST api connection token for your Dynatrace Managed system (see your

Admin andor Dynatrace API - Authentication)

Mediator creation and configuration

To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new origin

Press lsquo+rsquo on the SOURCES column to add a new source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

37

Now provide the connectivity information for your Dynatrace Managed system

Enter your Dynatrace Managed credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

Dynatrace Managed environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to produce IBM PI CSV files

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

38

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Select the desired Dynatrace Managed metric sources

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

39

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 18: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

18

Accessing the Mediation system

Once installed you can interact with the Mediation system via web-browser By default the

Mediation system listens on port 7457 so for example point your web-browser to

httpltservergt7457

Weaver Pressing lsquoWEAVERrsquo brings you back to the above screen You select the lsquo+rsquo and a list

of source options will appear Any 3rd party tool listed can be selected to begin the configuration process In a similar fashion Destinations can also be configured by selecting the plus icon in the Destinations section

Dashboard Pressing lsquoDASHBOARDrsquo while cause latency measurements (the duration of last collection for each destination mediator) for each stream to be presented

Configuration Pressing lsquoDASHBOARDrsquo shows configuration information for the platform

including particularly log levels

System Logs Pressing lsquoSYSTEM LOGSrsquo presents the latest log entries (the full set of logs and be viewed in the relevant log directory which is described later in this document)

In the subsequent Mediator-specific sections we will give specific examples of

Mediator-gtDistributor Each example will specify a specific kind of distributor ie CSV or REST

You are free to use either distributor in each case

The initial step in each configuration case is to press lsquo+rsquo on the SOURCES and then you should

be presented with a list of available Mediator source types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

19

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

20

AppDynamics Mediator

Overview

Extraction Approach

Metric data is extracted using the AppDynamics REST API

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

AppD Port Defaults

REST 443

Mediator creation and configuration

To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your AppDynamics source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

21

Select Destination

Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single

file stream emerges from each Destination and we donrsquot generally want to mix data from

different Collector instances

Give this destination a unique name This name will also be used as a prefix for the

AppDynamics CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

22

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from AppDynamics is based upon selecting Applications Tiers and metric

paths SWP will extract all metrics for the supplied paths

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

23

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)

when data for the next time interval arrives

You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open

for the specified number of periods (as defined by the Maxium Time Window Per File) This is

useful when you want to wait for late data The trade-off is that the files are only made available

to PI after that lag period - thereby increasing end to end latency

For example if you had 5min files and set the max lag to 2 yoursquod have current file open and

two older files leading to up to 15 min latency

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

24

AWS Cloudwatch Mediator

Overview

This mediator extracts Metric data from AWS Cloudwatch system

Extraction Approach

Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference

Pre-requisites

You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users

The Endpoint regions are shown in the following table For the subsequent SWP configuration

you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account

has access to metric data

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

25

T

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

26

Mediator creation and configuration

To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the Source column to add a new origin

Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for AWS Cloudwatch account

Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least

one Region Multiple regions can be entered by separating each with a comma

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the AWS

CloudWatch environment)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

27

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

28

Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics

in those categories SWP will extract all metrics for those selected metric tables For IBM PI

only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are

selected

The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide

convenient resource groupings If you want to scope your metric extraction to particular tagged

instances specify those tags here as comma separated tagname=tagvalue pairs

For more details on AWS tagging see the AWS documentation

The optimum settings for the other configuration values depend on your AWS accountdata

availability etc The defaults should be sufficient to get you operational

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

29

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

30

CA APM (Introscope) Mediator

Overview

CA APM was formerly known as Wily Introscope

Metric extraction from CA APM is based upon configuring the system to extract data for specific

Hosts Metric data will be collected for all Agents and Processes on each selected host

Extraction Approach

Metric data is extracted using a JDBC interface

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

CA APM Port Defaults

JDBC 5001

Note

The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used

Mediator creation and configuration

To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

and then press lsquo+rsquo on the Sources column to add a new sources

Provide a name for the new source instance

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

31

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your CA APM DB

The Metadata Window Width and Latency control time window of metric data which is

extracted to determine the available monitored hosts for selection

Note It is also possible to configure selection to use regexp expressions which are used directly

in the JDBC queries sent to CA APM Depending on whether or not you check the regexp

button you will be presented with a list of Agents(Hosts) to select from or be required to

provide regexps for Agents and Metric selection - see next section

The TimeZone is that of the CA APM server This must be configured to ensure the metric data

is correctly mapped to UTC

Enter the connection information for the CA APM Host

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the CA

APM host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

32

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name

Give this destination a unique name This name will also be used as a prefix for the caapm CSV

files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

33

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from CA APM is based upon selecting the hosts and SWP will extract all

metrics for that host

Note If you had chosen to configure via regexp the list of hosts panel will be empty

You must provide a set of comma separated regexp pairs - one pair per line in the RegExp

panel eg

User

These are regexps for CA APM Agents and Metrics and are used directly in the queries eg

select from metric_data where agent= and metric=User

See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

34

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Note Sometimes depending on the responsiveness of the CA-APM system it may take a few

seconds for the list of hosts to appear Clicking in the panel often helps with refresh

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

35

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

36

Dynatrace Managed

Overview

This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api

Extraction Approach

Pre-requisites

The hostport and REST api connection token for your Dynatrace Managed system (see your

Admin andor Dynatrace API - Authentication)

Mediator creation and configuration

To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new origin

Press lsquo+rsquo on the SOURCES column to add a new source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

37

Now provide the connectivity information for your Dynatrace Managed system

Enter your Dynatrace Managed credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

Dynatrace Managed environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to produce IBM PI CSV files

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

38

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Select the desired Dynatrace Managed metric sources

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

39

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 19: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

19

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

20

AppDynamics Mediator

Overview

Extraction Approach

Metric data is extracted using the AppDynamics REST API

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

AppD Port Defaults

REST 443

Mediator creation and configuration

To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your AppDynamics source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

21

Select Destination

Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single

file stream emerges from each Destination and we donrsquot generally want to mix data from

different Collector instances

Give this destination a unique name This name will also be used as a prefix for the

AppDynamics CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

22

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from AppDynamics is based upon selecting Applications Tiers and metric

paths SWP will extract all metrics for the supplied paths

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

23

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)

when data for the next time interval arrives

You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open

for the specified number of periods (as defined by the Maxium Time Window Per File) This is

useful when you want to wait for late data The trade-off is that the files are only made available

to PI after that lag period - thereby increasing end to end latency

For example if you had 5min files and set the max lag to 2 yoursquod have current file open and

two older files leading to up to 15 min latency

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

24

AWS Cloudwatch Mediator

Overview

This mediator extracts Metric data from AWS Cloudwatch system

Extraction Approach

Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference

Pre-requisites

You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users

The Endpoint regions are shown in the following table For the subsequent SWP configuration

you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account

has access to metric data

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

25

T

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

26

Mediator creation and configuration

To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the Source column to add a new origin

Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for AWS Cloudwatch account

Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least

one Region Multiple regions can be entered by separating each with a comma

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the AWS

CloudWatch environment)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

27

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

28

Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics

in those categories SWP will extract all metrics for those selected metric tables For IBM PI

only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are

selected

The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide

convenient resource groupings If you want to scope your metric extraction to particular tagged

instances specify those tags here as comma separated tagname=tagvalue pairs

For more details on AWS tagging see the AWS documentation

The optimum settings for the other configuration values depend on your AWS accountdata

availability etc The defaults should be sufficient to get you operational

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

29

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

30

CA APM (Introscope) Mediator

Overview

CA APM was formerly known as Wily Introscope

Metric extraction from CA APM is based upon configuring the system to extract data for specific

Hosts Metric data will be collected for all Agents and Processes on each selected host

Extraction Approach

Metric data is extracted using a JDBC interface

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

CA APM Port Defaults

JDBC 5001

Note

The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used

Mediator creation and configuration

To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

and then press lsquo+rsquo on the Sources column to add a new sources

Provide a name for the new source instance

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

31

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your CA APM DB

The Metadata Window Width and Latency control time window of metric data which is

extracted to determine the available monitored hosts for selection

Note It is also possible to configure selection to use regexp expressions which are used directly

in the JDBC queries sent to CA APM Depending on whether or not you check the regexp

button you will be presented with a list of Agents(Hosts) to select from or be required to

provide regexps for Agents and Metric selection - see next section

The TimeZone is that of the CA APM server This must be configured to ensure the metric data

is correctly mapped to UTC

Enter the connection information for the CA APM Host

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the CA

APM host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

32

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name

Give this destination a unique name This name will also be used as a prefix for the caapm CSV

files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

33

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from CA APM is based upon selecting the hosts and SWP will extract all

metrics for that host

Note If you had chosen to configure via regexp the list of hosts panel will be empty

You must provide a set of comma separated regexp pairs - one pair per line in the RegExp

panel eg

User

These are regexps for CA APM Agents and Metrics and are used directly in the queries eg

select from metric_data where agent= and metric=User

See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

34

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Note Sometimes depending on the responsiveness of the CA-APM system it may take a few

seconds for the list of hosts to appear Clicking in the panel often helps with refresh

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

35

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

36

Dynatrace Managed

Overview

This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api

Extraction Approach

Pre-requisites

The hostport and REST api connection token for your Dynatrace Managed system (see your

Admin andor Dynatrace API - Authentication)

Mediator creation and configuration

To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new origin

Press lsquo+rsquo on the SOURCES column to add a new source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

37

Now provide the connectivity information for your Dynatrace Managed system

Enter your Dynatrace Managed credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

Dynatrace Managed environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to produce IBM PI CSV files

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

38

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Select the desired Dynatrace Managed metric sources

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

39

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 20: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

20

AppDynamics Mediator

Overview

Extraction Approach

Metric data is extracted using the AppDynamics REST API

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

AppD Port Defaults

REST 443

Mediator creation and configuration

To create a new AppDynamice to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your AppDynamics source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

21

Select Destination

Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single

file stream emerges from each Destination and we donrsquot generally want to mix data from

different Collector instances

Give this destination a unique name This name will also be used as a prefix for the

AppDynamics CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

22

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from AppDynamics is based upon selecting Applications Tiers and metric

paths SWP will extract all metrics for the supplied paths

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

23

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)

when data for the next time interval arrives

You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open

for the specified number of periods (as defined by the Maxium Time Window Per File) This is

useful when you want to wait for late data The trade-off is that the files are only made available

to PI after that lag period - thereby increasing end to end latency

For example if you had 5min files and set the max lag to 2 yoursquod have current file open and

two older files leading to up to 15 min latency

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

24

AWS Cloudwatch Mediator

Overview

This mediator extracts Metric data from AWS Cloudwatch system

Extraction Approach

Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference

Pre-requisites

You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users

The Endpoint regions are shown in the following table For the subsequent SWP configuration

you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account

has access to metric data

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

25

T

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

26

Mediator creation and configuration

To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the Source column to add a new origin

Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for AWS Cloudwatch account

Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least

one Region Multiple regions can be entered by separating each with a comma

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the AWS

CloudWatch environment)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

27

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

28

Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics

in those categories SWP will extract all metrics for those selected metric tables For IBM PI

only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are

selected

The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide

convenient resource groupings If you want to scope your metric extraction to particular tagged

instances specify those tags here as comma separated tagname=tagvalue pairs

For more details on AWS tagging see the AWS documentation

The optimum settings for the other configuration values depend on your AWS accountdata

availability etc The defaults should be sufficient to get you operational

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

29

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

30

CA APM (Introscope) Mediator

Overview

CA APM was formerly known as Wily Introscope

Metric extraction from CA APM is based upon configuring the system to extract data for specific

Hosts Metric data will be collected for all Agents and Processes on each selected host

Extraction Approach

Metric data is extracted using a JDBC interface

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

CA APM Port Defaults

JDBC 5001

Note

The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used

Mediator creation and configuration

To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

and then press lsquo+rsquo on the Sources column to add a new sources

Provide a name for the new source instance

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

31

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your CA APM DB

The Metadata Window Width and Latency control time window of metric data which is

extracted to determine the available monitored hosts for selection

Note It is also possible to configure selection to use regexp expressions which are used directly

in the JDBC queries sent to CA APM Depending on whether or not you check the regexp

button you will be presented with a list of Agents(Hosts) to select from or be required to

provide regexps for Agents and Metric selection - see next section

The TimeZone is that of the CA APM server This must be configured to ensure the metric data

is correctly mapped to UTC

Enter the connection information for the CA APM Host

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the CA

APM host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

32

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name

Give this destination a unique name This name will also be used as a prefix for the caapm CSV

files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

33

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from CA APM is based upon selecting the hosts and SWP will extract all

metrics for that host

Note If you had chosen to configure via regexp the list of hosts panel will be empty

You must provide a set of comma separated regexp pairs - one pair per line in the RegExp

panel eg

User

These are regexps for CA APM Agents and Metrics and are used directly in the queries eg

select from metric_data where agent= and metric=User

See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

34

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Note Sometimes depending on the responsiveness of the CA-APM system it may take a few

seconds for the list of hosts to appear Clicking in the panel often helps with refresh

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

35

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

36

Dynatrace Managed

Overview

This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api

Extraction Approach

Pre-requisites

The hostport and REST api connection token for your Dynatrace Managed system (see your

Admin andor Dynatrace API - Authentication)

Mediator creation and configuration

To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new origin

Press lsquo+rsquo on the SOURCES column to add a new source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

37

Now provide the connectivity information for your Dynatrace Managed system

Enter your Dynatrace Managed credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

Dynatrace Managed environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to produce IBM PI CSV files

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

38

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Select the desired Dynatrace Managed metric sources

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

39

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 21: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

21

Select Destination

Now select the destination You should almost always select lsquoNEW DESTINATIONrsquo as a single

file stream emerges from each Destination and we donrsquot generally want to mix data from

different Collector instances

Give this destination a unique name This name will also be used as a prefix for the

AppDynamics CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

22

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from AppDynamics is based upon selecting Applications Tiers and metric

paths SWP will extract all metrics for the supplied paths

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

23

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)

when data for the next time interval arrives

You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open

for the specified number of periods (as defined by the Maxium Time Window Per File) This is

useful when you want to wait for late data The trade-off is that the files are only made available

to PI after that lag period - thereby increasing end to end latency

For example if you had 5min files and set the max lag to 2 yoursquod have current file open and

two older files leading to up to 15 min latency

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

24

AWS Cloudwatch Mediator

Overview

This mediator extracts Metric data from AWS Cloudwatch system

Extraction Approach

Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference

Pre-requisites

You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users

The Endpoint regions are shown in the following table For the subsequent SWP configuration

you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account

has access to metric data

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

25

T

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

26

Mediator creation and configuration

To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the Source column to add a new origin

Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for AWS Cloudwatch account

Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least

one Region Multiple regions can be entered by separating each with a comma

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the AWS

CloudWatch environment)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

27

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

28

Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics

in those categories SWP will extract all metrics for those selected metric tables For IBM PI

only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are

selected

The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide

convenient resource groupings If you want to scope your metric extraction to particular tagged

instances specify those tags here as comma separated tagname=tagvalue pairs

For more details on AWS tagging see the AWS documentation

The optimum settings for the other configuration values depend on your AWS accountdata

availability etc The defaults should be sufficient to get you operational

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

29

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

30

CA APM (Introscope) Mediator

Overview

CA APM was formerly known as Wily Introscope

Metric extraction from CA APM is based upon configuring the system to extract data for specific

Hosts Metric data will be collected for all Agents and Processes on each selected host

Extraction Approach

Metric data is extracted using a JDBC interface

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

CA APM Port Defaults

JDBC 5001

Note

The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used

Mediator creation and configuration

To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

and then press lsquo+rsquo on the Sources column to add a new sources

Provide a name for the new source instance

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

31

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your CA APM DB

The Metadata Window Width and Latency control time window of metric data which is

extracted to determine the available monitored hosts for selection

Note It is also possible to configure selection to use regexp expressions which are used directly

in the JDBC queries sent to CA APM Depending on whether or not you check the regexp

button you will be presented with a list of Agents(Hosts) to select from or be required to

provide regexps for Agents and Metric selection - see next section

The TimeZone is that of the CA APM server This must be configured to ensure the metric data

is correctly mapped to UTC

Enter the connection information for the CA APM Host

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the CA

APM host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

32

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name

Give this destination a unique name This name will also be used as a prefix for the caapm CSV

files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

33

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from CA APM is based upon selecting the hosts and SWP will extract all

metrics for that host

Note If you had chosen to configure via regexp the list of hosts panel will be empty

You must provide a set of comma separated regexp pairs - one pair per line in the RegExp

panel eg

User

These are regexps for CA APM Agents and Metrics and are used directly in the queries eg

select from metric_data where agent= and metric=User

See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

34

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Note Sometimes depending on the responsiveness of the CA-APM system it may take a few

seconds for the list of hosts to appear Clicking in the panel often helps with refresh

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

35

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

36

Dynatrace Managed

Overview

This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api

Extraction Approach

Pre-requisites

The hostport and REST api connection token for your Dynatrace Managed system (see your

Admin andor Dynatrace API - Authentication)

Mediator creation and configuration

To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new origin

Press lsquo+rsquo on the SOURCES column to add a new source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

37

Now provide the connectivity information for your Dynatrace Managed system

Enter your Dynatrace Managed credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

Dynatrace Managed environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to produce IBM PI CSV files

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

38

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Select the desired Dynatrace Managed metric sources

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

39

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 22: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

22

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from AppDynamics is based upon selecting Applications Tiers and metric

paths SWP will extract all metrics for the supplied paths

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

23

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)

when data for the next time interval arrives

You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open

for the specified number of periods (as defined by the Maxium Time Window Per File) This is

useful when you want to wait for late data The trade-off is that the files are only made available

to PI after that lag period - thereby increasing end to end latency

For example if you had 5min files and set the max lag to 2 yoursquod have current file open and

two older files leading to up to 15 min latency

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

24

AWS Cloudwatch Mediator

Overview

This mediator extracts Metric data from AWS Cloudwatch system

Extraction Approach

Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference

Pre-requisites

You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users

The Endpoint regions are shown in the following table For the subsequent SWP configuration

you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account

has access to metric data

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

25

T

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

26

Mediator creation and configuration

To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the Source column to add a new origin

Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for AWS Cloudwatch account

Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least

one Region Multiple regions can be entered by separating each with a comma

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the AWS

CloudWatch environment)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

27

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

28

Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics

in those categories SWP will extract all metrics for those selected metric tables For IBM PI

only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are

selected

The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide

convenient resource groupings If you want to scope your metric extraction to particular tagged

instances specify those tags here as comma separated tagname=tagvalue pairs

For more details on AWS tagging see the AWS documentation

The optimum settings for the other configuration values depend on your AWS accountdata

availability etc The defaults should be sufficient to get you operational

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

29

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

30

CA APM (Introscope) Mediator

Overview

CA APM was formerly known as Wily Introscope

Metric extraction from CA APM is based upon configuring the system to extract data for specific

Hosts Metric data will be collected for all Agents and Processes on each selected host

Extraction Approach

Metric data is extracted using a JDBC interface

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

CA APM Port Defaults

JDBC 5001

Note

The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used

Mediator creation and configuration

To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

and then press lsquo+rsquo on the Sources column to add a new sources

Provide a name for the new source instance

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

31

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your CA APM DB

The Metadata Window Width and Latency control time window of metric data which is

extracted to determine the available monitored hosts for selection

Note It is also possible to configure selection to use regexp expressions which are used directly

in the JDBC queries sent to CA APM Depending on whether or not you check the regexp

button you will be presented with a list of Agents(Hosts) to select from or be required to

provide regexps for Agents and Metric selection - see next section

The TimeZone is that of the CA APM server This must be configured to ensure the metric data

is correctly mapped to UTC

Enter the connection information for the CA APM Host

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the CA

APM host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

32

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name

Give this destination a unique name This name will also be used as a prefix for the caapm CSV

files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

33

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from CA APM is based upon selecting the hosts and SWP will extract all

metrics for that host

Note If you had chosen to configure via regexp the list of hosts panel will be empty

You must provide a set of comma separated regexp pairs - one pair per line in the RegExp

panel eg

User

These are regexps for CA APM Agents and Metrics and are used directly in the queries eg

select from metric_data where agent= and metric=User

See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

34

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Note Sometimes depending on the responsiveness of the CA-APM system it may take a few

seconds for the list of hosts to appear Clicking in the panel often helps with refresh

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

35

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

36

Dynatrace Managed

Overview

This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api

Extraction Approach

Pre-requisites

The hostport and REST api connection token for your Dynatrace Managed system (see your

Admin andor Dynatrace API - Authentication)

Mediator creation and configuration

To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new origin

Press lsquo+rsquo on the SOURCES column to add a new source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

37

Now provide the connectivity information for your Dynatrace Managed system

Enter your Dynatrace Managed credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

Dynatrace Managed environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to produce IBM PI CSV files

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

38

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Select the desired Dynatrace Managed metric sources

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

39

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 23: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

23

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also closed and lsquorollrsquoed (be renamed to PI standard CSV file names)

when data for the next time interval arrives

You can also configure a lsquoMax File Lag Periodsrsquo setting which will allow files to remain open

for the specified number of periods (as defined by the Maxium Time Window Per File) This is

useful when you want to wait for late data The trade-off is that the files are only made available

to PI after that lag period - thereby increasing end to end latency

For example if you had 5min files and set the max lag to 2 yoursquod have current file open and

two older files leading to up to 15 min latency

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

24

AWS Cloudwatch Mediator

Overview

This mediator extracts Metric data from AWS Cloudwatch system

Extraction Approach

Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference

Pre-requisites

You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users

The Endpoint regions are shown in the following table For the subsequent SWP configuration

you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account

has access to metric data

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

25

T

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

26

Mediator creation and configuration

To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the Source column to add a new origin

Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for AWS Cloudwatch account

Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least

one Region Multiple regions can be entered by separating each with a comma

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the AWS

CloudWatch environment)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

27

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

28

Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics

in those categories SWP will extract all metrics for those selected metric tables For IBM PI

only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are

selected

The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide

convenient resource groupings If you want to scope your metric extraction to particular tagged

instances specify those tags here as comma separated tagname=tagvalue pairs

For more details on AWS tagging see the AWS documentation

The optimum settings for the other configuration values depend on your AWS accountdata

availability etc The defaults should be sufficient to get you operational

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

29

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

30

CA APM (Introscope) Mediator

Overview

CA APM was formerly known as Wily Introscope

Metric extraction from CA APM is based upon configuring the system to extract data for specific

Hosts Metric data will be collected for all Agents and Processes on each selected host

Extraction Approach

Metric data is extracted using a JDBC interface

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

CA APM Port Defaults

JDBC 5001

Note

The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used

Mediator creation and configuration

To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

and then press lsquo+rsquo on the Sources column to add a new sources

Provide a name for the new source instance

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

31

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your CA APM DB

The Metadata Window Width and Latency control time window of metric data which is

extracted to determine the available monitored hosts for selection

Note It is also possible to configure selection to use regexp expressions which are used directly

in the JDBC queries sent to CA APM Depending on whether or not you check the regexp

button you will be presented with a list of Agents(Hosts) to select from or be required to

provide regexps for Agents and Metric selection - see next section

The TimeZone is that of the CA APM server This must be configured to ensure the metric data

is correctly mapped to UTC

Enter the connection information for the CA APM Host

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the CA

APM host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

32

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name

Give this destination a unique name This name will also be used as a prefix for the caapm CSV

files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

33

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from CA APM is based upon selecting the hosts and SWP will extract all

metrics for that host

Note If you had chosen to configure via regexp the list of hosts panel will be empty

You must provide a set of comma separated regexp pairs - one pair per line in the RegExp

panel eg

User

These are regexps for CA APM Agents and Metrics and are used directly in the queries eg

select from metric_data where agent= and metric=User

See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

34

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Note Sometimes depending on the responsiveness of the CA-APM system it may take a few

seconds for the list of hosts to appear Clicking in the panel often helps with refresh

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

35

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

36

Dynatrace Managed

Overview

This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api

Extraction Approach

Pre-requisites

The hostport and REST api connection token for your Dynatrace Managed system (see your

Admin andor Dynatrace API - Authentication)

Mediator creation and configuration

To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new origin

Press lsquo+rsquo on the SOURCES column to add a new source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

37

Now provide the connectivity information for your Dynatrace Managed system

Enter your Dynatrace Managed credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

Dynatrace Managed environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to produce IBM PI CSV files

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

38

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Select the desired Dynatrace Managed metric sources

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

39

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 24: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

24

AWS Cloudwatch Mediator

Overview

This mediator extracts Metric data from AWS Cloudwatch system

Extraction Approach

Metric data is extracted using a REST interface Further info can be found here Amazon Cloud Watch API Reference

Pre-requisites

You must have an AWS lsquoAccess Keyrsquo and lsquoSecret Keyrsquo and at least one specified lsquoEndpoint Regionrsquo identified See AWS credentials and keys for details on Managing Access Keys for IAM users

The Endpoint regions are shown in the following table For the subsequent SWP configuration

you will need to identify at least one lsquoRegionrsquo (column 2) from this table for which your account

has access to metric data

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

25

T

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

26

Mediator creation and configuration

To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the Source column to add a new origin

Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for AWS Cloudwatch account

Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least

one Region Multiple regions can be entered by separating each with a comma

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the AWS

CloudWatch environment)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

27

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

28

Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics

in those categories SWP will extract all metrics for those selected metric tables For IBM PI

only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are

selected

The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide

convenient resource groupings If you want to scope your metric extraction to particular tagged

instances specify those tags here as comma separated tagname=tagvalue pairs

For more details on AWS tagging see the AWS documentation

The optimum settings for the other configuration values depend on your AWS accountdata

availability etc The defaults should be sufficient to get you operational

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

29

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

30

CA APM (Introscope) Mediator

Overview

CA APM was formerly known as Wily Introscope

Metric extraction from CA APM is based upon configuring the system to extract data for specific

Hosts Metric data will be collected for all Agents and Processes on each selected host

Extraction Approach

Metric data is extracted using a JDBC interface

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

CA APM Port Defaults

JDBC 5001

Note

The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used

Mediator creation and configuration

To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

and then press lsquo+rsquo on the Sources column to add a new sources

Provide a name for the new source instance

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

31

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your CA APM DB

The Metadata Window Width and Latency control time window of metric data which is

extracted to determine the available monitored hosts for selection

Note It is also possible to configure selection to use regexp expressions which are used directly

in the JDBC queries sent to CA APM Depending on whether or not you check the regexp

button you will be presented with a list of Agents(Hosts) to select from or be required to

provide regexps for Agents and Metric selection - see next section

The TimeZone is that of the CA APM server This must be configured to ensure the metric data

is correctly mapped to UTC

Enter the connection information for the CA APM Host

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the CA

APM host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

32

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name

Give this destination a unique name This name will also be used as a prefix for the caapm CSV

files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

33

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from CA APM is based upon selecting the hosts and SWP will extract all

metrics for that host

Note If you had chosen to configure via regexp the list of hosts panel will be empty

You must provide a set of comma separated regexp pairs - one pair per line in the RegExp

panel eg

User

These are regexps for CA APM Agents and Metrics and are used directly in the queries eg

select from metric_data where agent= and metric=User

See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

34

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Note Sometimes depending on the responsiveness of the CA-APM system it may take a few

seconds for the list of hosts to appear Clicking in the panel often helps with refresh

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

35

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

36

Dynatrace Managed

Overview

This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api

Extraction Approach

Pre-requisites

The hostport and REST api connection token for your Dynatrace Managed system (see your

Admin andor Dynatrace API - Authentication)

Mediator creation and configuration

To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new origin

Press lsquo+rsquo on the SOURCES column to add a new source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

37

Now provide the connectivity information for your Dynatrace Managed system

Enter your Dynatrace Managed credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

Dynatrace Managed environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to produce IBM PI CSV files

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

38

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Select the desired Dynatrace Managed metric sources

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

39

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 25: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

25

T

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

26

Mediator creation and configuration

To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the Source column to add a new origin

Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for AWS Cloudwatch account

Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least

one Region Multiple regions can be entered by separating each with a comma

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the AWS

CloudWatch environment)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

27

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

28

Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics

in those categories SWP will extract all metrics for those selected metric tables For IBM PI

only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are

selected

The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide

convenient resource groupings If you want to scope your metric extraction to particular tagged

instances specify those tags here as comma separated tagname=tagvalue pairs

For more details on AWS tagging see the AWS documentation

The optimum settings for the other configuration values depend on your AWS accountdata

availability etc The defaults should be sufficient to get you operational

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

29

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

30

CA APM (Introscope) Mediator

Overview

CA APM was formerly known as Wily Introscope

Metric extraction from CA APM is based upon configuring the system to extract data for specific

Hosts Metric data will be collected for all Agents and Processes on each selected host

Extraction Approach

Metric data is extracted using a JDBC interface

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

CA APM Port Defaults

JDBC 5001

Note

The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used

Mediator creation and configuration

To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

and then press lsquo+rsquo on the Sources column to add a new sources

Provide a name for the new source instance

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

31

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your CA APM DB

The Metadata Window Width and Latency control time window of metric data which is

extracted to determine the available monitored hosts for selection

Note It is also possible to configure selection to use regexp expressions which are used directly

in the JDBC queries sent to CA APM Depending on whether or not you check the regexp

button you will be presented with a list of Agents(Hosts) to select from or be required to

provide regexps for Agents and Metric selection - see next section

The TimeZone is that of the CA APM server This must be configured to ensure the metric data

is correctly mapped to UTC

Enter the connection information for the CA APM Host

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the CA

APM host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

32

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name

Give this destination a unique name This name will also be used as a prefix for the caapm CSV

files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

33

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from CA APM is based upon selecting the hosts and SWP will extract all

metrics for that host

Note If you had chosen to configure via regexp the list of hosts panel will be empty

You must provide a set of comma separated regexp pairs - one pair per line in the RegExp

panel eg

User

These are regexps for CA APM Agents and Metrics and are used directly in the queries eg

select from metric_data where agent= and metric=User

See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

34

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Note Sometimes depending on the responsiveness of the CA-APM system it may take a few

seconds for the list of hosts to appear Clicking in the panel often helps with refresh

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

35

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

36

Dynatrace Managed

Overview

This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api

Extraction Approach

Pre-requisites

The hostport and REST api connection token for your Dynatrace Managed system (see your

Admin andor Dynatrace API - Authentication)

Mediator creation and configuration

To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new origin

Press lsquo+rsquo on the SOURCES column to add a new source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

37

Now provide the connectivity information for your Dynatrace Managed system

Enter your Dynatrace Managed credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

Dynatrace Managed environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to produce IBM PI CSV files

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

38

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Select the desired Dynatrace Managed metric sources

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

39

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 26: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

26

Mediator creation and configuration

To create a new AWS Cloudwatch to PI REST stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the Source column to add a new origin

Then click on lsquoCREATE NEW Sourcersquo and select lsquoAWS CloudWatchrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for AWS Cloudwatch account

Enter the Access Key and Secret Key information (see pre-req section above) Also enter at least

one Region Multiple regions can be entered by separating each with a comma

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the AWS

CloudWatch environment)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

27

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

28

Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics

in those categories SWP will extract all metrics for those selected metric tables For IBM PI

only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are

selected

The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide

convenient resource groupings If you want to scope your metric extraction to particular tagged

instances specify those tags here as comma separated tagname=tagvalue pairs

For more details on AWS tagging see the AWS documentation

The optimum settings for the other configuration values depend on your AWS accountdata

availability etc The defaults should be sufficient to get you operational

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

29

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

30

CA APM (Introscope) Mediator

Overview

CA APM was formerly known as Wily Introscope

Metric extraction from CA APM is based upon configuring the system to extract data for specific

Hosts Metric data will be collected for all Agents and Processes on each selected host

Extraction Approach

Metric data is extracted using a JDBC interface

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

CA APM Port Defaults

JDBC 5001

Note

The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used

Mediator creation and configuration

To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

and then press lsquo+rsquo on the Sources column to add a new sources

Provide a name for the new source instance

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

31

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your CA APM DB

The Metadata Window Width and Latency control time window of metric data which is

extracted to determine the available monitored hosts for selection

Note It is also possible to configure selection to use regexp expressions which are used directly

in the JDBC queries sent to CA APM Depending on whether or not you check the regexp

button you will be presented with a list of Agents(Hosts) to select from or be required to

provide regexps for Agents and Metric selection - see next section

The TimeZone is that of the CA APM server This must be configured to ensure the metric data

is correctly mapped to UTC

Enter the connection information for the CA APM Host

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the CA

APM host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

32

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name

Give this destination a unique name This name will also be used as a prefix for the caapm CSV

files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

33

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from CA APM is based upon selecting the hosts and SWP will extract all

metrics for that host

Note If you had chosen to configure via regexp the list of hosts panel will be empty

You must provide a set of comma separated regexp pairs - one pair per line in the RegExp

panel eg

User

These are regexps for CA APM Agents and Metrics and are used directly in the queries eg

select from metric_data where agent= and metric=User

See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

34

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Note Sometimes depending on the responsiveness of the CA-APM system it may take a few

seconds for the list of hosts to appear Clicking in the panel often helps with refresh

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

35

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

36

Dynatrace Managed

Overview

This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api

Extraction Approach

Pre-requisites

The hostport and REST api connection token for your Dynatrace Managed system (see your

Admin andor Dynatrace API - Authentication)

Mediator creation and configuration

To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new origin

Press lsquo+rsquo on the SOURCES column to add a new source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

37

Now provide the connectivity information for your Dynatrace Managed system

Enter your Dynatrace Managed credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

Dynatrace Managed environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to produce IBM PI CSV files

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

38

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Select the desired Dynatrace Managed metric sources

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

39

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 27: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

27

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

28

Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics

in those categories SWP will extract all metrics for those selected metric tables For IBM PI

only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are

selected

The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide

convenient resource groupings If you want to scope your metric extraction to particular tagged

instances specify those tags here as comma separated tagname=tagvalue pairs

For more details on AWS tagging see the AWS documentation

The optimum settings for the other configuration values depend on your AWS accountdata

availability etc The defaults should be sufficient to get you operational

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

29

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

30

CA APM (Introscope) Mediator

Overview

CA APM was formerly known as Wily Introscope

Metric extraction from CA APM is based upon configuring the system to extract data for specific

Hosts Metric data will be collected for all Agents and Processes on each selected host

Extraction Approach

Metric data is extracted using a JDBC interface

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

CA APM Port Defaults

JDBC 5001

Note

The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used

Mediator creation and configuration

To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

and then press lsquo+rsquo on the Sources column to add a new sources

Provide a name for the new source instance

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

31

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your CA APM DB

The Metadata Window Width and Latency control time window of metric data which is

extracted to determine the available monitored hosts for selection

Note It is also possible to configure selection to use regexp expressions which are used directly

in the JDBC queries sent to CA APM Depending on whether or not you check the regexp

button you will be presented with a list of Agents(Hosts) to select from or be required to

provide regexps for Agents and Metric selection - see next section

The TimeZone is that of the CA APM server This must be configured to ensure the metric data

is correctly mapped to UTC

Enter the connection information for the CA APM Host

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the CA

APM host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

32

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name

Give this destination a unique name This name will also be used as a prefix for the caapm CSV

files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

33

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from CA APM is based upon selecting the hosts and SWP will extract all

metrics for that host

Note If you had chosen to configure via regexp the list of hosts panel will be empty

You must provide a set of comma separated regexp pairs - one pair per line in the RegExp

panel eg

User

These are regexps for CA APM Agents and Metrics and are used directly in the queries eg

select from metric_data where agent= and metric=User

See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

34

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Note Sometimes depending on the responsiveness of the CA-APM system it may take a few

seconds for the list of hosts to appear Clicking in the panel often helps with refresh

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

35

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

36

Dynatrace Managed

Overview

This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api

Extraction Approach

Pre-requisites

The hostport and REST api connection token for your Dynatrace Managed system (see your

Admin andor Dynatrace API - Authentication)

Mediator creation and configuration

To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new origin

Press lsquo+rsquo on the SOURCES column to add a new source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

37

Now provide the connectivity information for your Dynatrace Managed system

Enter your Dynatrace Managed credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

Dynatrace Managed environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to produce IBM PI CSV files

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

38

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Select the desired Dynatrace Managed metric sources

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

39

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 28: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

28

Metric selection from AWS CloudWatch is from a set of metric categories and then from metrics

in those categories SWP will extract all metrics for those selected metric tables For IBM PI

only the lsquoPerformance Metricsrsquo are of interest so ensure that all the lsquoPerformance Metricsrsquo are

selected

The other selection is via lsquoInstance Tagsrsquo These tags are an AWS concept used to provide

convenient resource groupings If you want to scope your metric extraction to particular tagged

instances specify those tags here as comma separated tagname=tagvalue pairs

For more details on AWS tagging see the AWS documentation

The optimum settings for the other configuration values depend on your AWS accountdata

availability etc The defaults should be sufficient to get you operational

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

29

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

30

CA APM (Introscope) Mediator

Overview

CA APM was formerly known as Wily Introscope

Metric extraction from CA APM is based upon configuring the system to extract data for specific

Hosts Metric data will be collected for all Agents and Processes on each selected host

Extraction Approach

Metric data is extracted using a JDBC interface

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

CA APM Port Defaults

JDBC 5001

Note

The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used

Mediator creation and configuration

To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

and then press lsquo+rsquo on the Sources column to add a new sources

Provide a name for the new source instance

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

31

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your CA APM DB

The Metadata Window Width and Latency control time window of metric data which is

extracted to determine the available monitored hosts for selection

Note It is also possible to configure selection to use regexp expressions which are used directly

in the JDBC queries sent to CA APM Depending on whether or not you check the regexp

button you will be presented with a list of Agents(Hosts) to select from or be required to

provide regexps for Agents and Metric selection - see next section

The TimeZone is that of the CA APM server This must be configured to ensure the metric data

is correctly mapped to UTC

Enter the connection information for the CA APM Host

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the CA

APM host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

32

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name

Give this destination a unique name This name will also be used as a prefix for the caapm CSV

files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

33

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from CA APM is based upon selecting the hosts and SWP will extract all

metrics for that host

Note If you had chosen to configure via regexp the list of hosts panel will be empty

You must provide a set of comma separated regexp pairs - one pair per line in the RegExp

panel eg

User

These are regexps for CA APM Agents and Metrics and are used directly in the queries eg

select from metric_data where agent= and metric=User

See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

34

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Note Sometimes depending on the responsiveness of the CA-APM system it may take a few

seconds for the list of hosts to appear Clicking in the panel often helps with refresh

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

35

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

36

Dynatrace Managed

Overview

This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api

Extraction Approach

Pre-requisites

The hostport and REST api connection token for your Dynatrace Managed system (see your

Admin andor Dynatrace API - Authentication)

Mediator creation and configuration

To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new origin

Press lsquo+rsquo on the SOURCES column to add a new source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

37

Now provide the connectivity information for your Dynatrace Managed system

Enter your Dynatrace Managed credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

Dynatrace Managed environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to produce IBM PI CSV files

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

38

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Select the desired Dynatrace Managed metric sources

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

39

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 29: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

29

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

30

CA APM (Introscope) Mediator

Overview

CA APM was formerly known as Wily Introscope

Metric extraction from CA APM is based upon configuring the system to extract data for specific

Hosts Metric data will be collected for all Agents and Processes on each selected host

Extraction Approach

Metric data is extracted using a JDBC interface

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

CA APM Port Defaults

JDBC 5001

Note

The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used

Mediator creation and configuration

To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

and then press lsquo+rsquo on the Sources column to add a new sources

Provide a name for the new source instance

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

31

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your CA APM DB

The Metadata Window Width and Latency control time window of metric data which is

extracted to determine the available monitored hosts for selection

Note It is also possible to configure selection to use regexp expressions which are used directly

in the JDBC queries sent to CA APM Depending on whether or not you check the regexp

button you will be presented with a list of Agents(Hosts) to select from or be required to

provide regexps for Agents and Metric selection - see next section

The TimeZone is that of the CA APM server This must be configured to ensure the metric data

is correctly mapped to UTC

Enter the connection information for the CA APM Host

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the CA

APM host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

32

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name

Give this destination a unique name This name will also be used as a prefix for the caapm CSV

files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

33

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from CA APM is based upon selecting the hosts and SWP will extract all

metrics for that host

Note If you had chosen to configure via regexp the list of hosts panel will be empty

You must provide a set of comma separated regexp pairs - one pair per line in the RegExp

panel eg

User

These are regexps for CA APM Agents and Metrics and are used directly in the queries eg

select from metric_data where agent= and metric=User

See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

34

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Note Sometimes depending on the responsiveness of the CA-APM system it may take a few

seconds for the list of hosts to appear Clicking in the panel often helps with refresh

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

35

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

36

Dynatrace Managed

Overview

This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api

Extraction Approach

Pre-requisites

The hostport and REST api connection token for your Dynatrace Managed system (see your

Admin andor Dynatrace API - Authentication)

Mediator creation and configuration

To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new origin

Press lsquo+rsquo on the SOURCES column to add a new source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

37

Now provide the connectivity information for your Dynatrace Managed system

Enter your Dynatrace Managed credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

Dynatrace Managed environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to produce IBM PI CSV files

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

38

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Select the desired Dynatrace Managed metric sources

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

39

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 30: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

30

CA APM (Introscope) Mediator

Overview

CA APM was formerly known as Wily Introscope

Metric extraction from CA APM is based upon configuring the system to extract data for specific

Hosts Metric data will be collected for all Agents and Processes on each selected host

Extraction Approach

Metric data is extracted using a JDBC interface

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

CA APM Port Defaults

JDBC 5001

Note

The AcceleratorMediators also has support for a CA APM integration This is present for legacy-support However for new integrations this SWPMediators based CA APM mediator should be used

Mediator creation and configuration

To create a new CA APM to PI CSV stream navigate to the SWP Weaver panel at

httpltmediationservergt7457

Select Source

and then press lsquo+rsquo on the Sources column to add a new sources

Provide a name for the new source instance

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

31

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your CA APM DB

The Metadata Window Width and Latency control time window of metric data which is

extracted to determine the available monitored hosts for selection

Note It is also possible to configure selection to use regexp expressions which are used directly

in the JDBC queries sent to CA APM Depending on whether or not you check the regexp

button you will be presented with a list of Agents(Hosts) to select from or be required to

provide regexps for Agents and Metric selection - see next section

The TimeZone is that of the CA APM server This must be configured to ensure the metric data

is correctly mapped to UTC

Enter the connection information for the CA APM Host

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the CA

APM host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

32

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name

Give this destination a unique name This name will also be used as a prefix for the caapm CSV

files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

33

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from CA APM is based upon selecting the hosts and SWP will extract all

metrics for that host

Note If you had chosen to configure via regexp the list of hosts panel will be empty

You must provide a set of comma separated regexp pairs - one pair per line in the RegExp

panel eg

User

These are regexps for CA APM Agents and Metrics and are used directly in the queries eg

select from metric_data where agent= and metric=User

See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

34

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Note Sometimes depending on the responsiveness of the CA-APM system it may take a few

seconds for the list of hosts to appear Clicking in the panel often helps with refresh

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

35

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

36

Dynatrace Managed

Overview

This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api

Extraction Approach

Pre-requisites

The hostport and REST api connection token for your Dynatrace Managed system (see your

Admin andor Dynatrace API - Authentication)

Mediator creation and configuration

To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new origin

Press lsquo+rsquo on the SOURCES column to add a new source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

37

Now provide the connectivity information for your Dynatrace Managed system

Enter your Dynatrace Managed credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

Dynatrace Managed environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to produce IBM PI CSV files

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

38

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Select the desired Dynatrace Managed metric sources

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

39

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 31: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

31

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your CA APM DB

The Metadata Window Width and Latency control time window of metric data which is

extracted to determine the available monitored hosts for selection

Note It is also possible to configure selection to use regexp expressions which are used directly

in the JDBC queries sent to CA APM Depending on whether or not you check the regexp

button you will be presented with a list of Agents(Hosts) to select from or be required to

provide regexps for Agents and Metric selection - see next section

The TimeZone is that of the CA APM server This must be configured to ensure the metric data

is correctly mapped to UTC

Enter the connection information for the CA APM Host

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the CA

APM host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

32

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name

Give this destination a unique name This name will also be used as a prefix for the caapm CSV

files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

33

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from CA APM is based upon selecting the hosts and SWP will extract all

metrics for that host

Note If you had chosen to configure via regexp the list of hosts panel will be empty

You must provide a set of comma separated regexp pairs - one pair per line in the RegExp

panel eg

User

These are regexps for CA APM Agents and Metrics and are used directly in the queries eg

select from metric_data where agent= and metric=User

See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

34

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Note Sometimes depending on the responsiveness of the CA-APM system it may take a few

seconds for the list of hosts to appear Clicking in the panel often helps with refresh

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

35

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

36

Dynatrace Managed

Overview

This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api

Extraction Approach

Pre-requisites

The hostport and REST api connection token for your Dynatrace Managed system (see your

Admin andor Dynatrace API - Authentication)

Mediator creation and configuration

To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new origin

Press lsquo+rsquo on the SOURCES column to add a new source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

37

Now provide the connectivity information for your Dynatrace Managed system

Enter your Dynatrace Managed credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

Dynatrace Managed environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to produce IBM PI CSV files

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

38

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Select the desired Dynatrace Managed metric sources

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

39

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 32: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

32

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name

Give this destination a unique name This name will also be used as a prefix for the caapm CSV

files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

33

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from CA APM is based upon selecting the hosts and SWP will extract all

metrics for that host

Note If you had chosen to configure via regexp the list of hosts panel will be empty

You must provide a set of comma separated regexp pairs - one pair per line in the RegExp

panel eg

User

These are regexps for CA APM Agents and Metrics and are used directly in the queries eg

select from metric_data where agent= and metric=User

See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

34

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Note Sometimes depending on the responsiveness of the CA-APM system it may take a few

seconds for the list of hosts to appear Clicking in the panel often helps with refresh

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

35

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

36

Dynatrace Managed

Overview

This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api

Extraction Approach

Pre-requisites

The hostport and REST api connection token for your Dynatrace Managed system (see your

Admin andor Dynatrace API - Authentication)

Mediator creation and configuration

To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new origin

Press lsquo+rsquo on the SOURCES column to add a new source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

37

Now provide the connectivity information for your Dynatrace Managed system

Enter your Dynatrace Managed credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

Dynatrace Managed environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to produce IBM PI CSV files

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

38

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Select the desired Dynatrace Managed metric sources

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

39

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 33: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

33

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from CA APM is based upon selecting the hosts and SWP will extract all

metrics for that host

Note If you had chosen to configure via regexp the list of hosts panel will be empty

You must provide a set of comma separated regexp pairs - one pair per line in the RegExp

panel eg

User

These are regexps for CA APM Agents and Metrics and are used directly in the queries eg

select from metric_data where agent= and metric=User

See httpsdocopscacomca-apm10-1enapi-referencejdbc-api for examplesdetails

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

34

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Note Sometimes depending on the responsiveness of the CA-APM system it may take a few

seconds for the list of hosts to appear Clicking in the panel often helps with refresh

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

35

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

36

Dynatrace Managed

Overview

This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api

Extraction Approach

Pre-requisites

The hostport and REST api connection token for your Dynatrace Managed system (see your

Admin andor Dynatrace API - Authentication)

Mediator creation and configuration

To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new origin

Press lsquo+rsquo on the SOURCES column to add a new source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

37

Now provide the connectivity information for your Dynatrace Managed system

Enter your Dynatrace Managed credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

Dynatrace Managed environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to produce IBM PI CSV files

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

38

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Select the desired Dynatrace Managed metric sources

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

39

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 34: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

34

Select the hosts you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Note Sometimes depending on the responsiveness of the CA-APM system it may take a few

seconds for the list of hosts to appear Clicking in the panel often helps with refresh

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

35

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

36

Dynatrace Managed

Overview

This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api

Extraction Approach

Pre-requisites

The hostport and REST api connection token for your Dynatrace Managed system (see your

Admin andor Dynatrace API - Authentication)

Mediator creation and configuration

To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new origin

Press lsquo+rsquo on the SOURCES column to add a new source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

37

Now provide the connectivity information for your Dynatrace Managed system

Enter your Dynatrace Managed credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

Dynatrace Managed environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to produce IBM PI CSV files

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

38

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Select the desired Dynatrace Managed metric sources

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

39

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 35: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

35

Configuring Distributors

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

36

Dynatrace Managed

Overview

This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api

Extraction Approach

Pre-requisites

The hostport and REST api connection token for your Dynatrace Managed system (see your

Admin andor Dynatrace API - Authentication)

Mediator creation and configuration

To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new origin

Press lsquo+rsquo on the SOURCES column to add a new source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

37

Now provide the connectivity information for your Dynatrace Managed system

Enter your Dynatrace Managed credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

Dynatrace Managed environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to produce IBM PI CSV files

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

38

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Select the desired Dynatrace Managed metric sources

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

39

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 36: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

36

Dynatrace Managed

Overview

This mediator extracts Metric data from lsquoDynatrace Managedlsquo using its REST api

Extraction Approach

Pre-requisites

The hostport and REST api connection token for your Dynatrace Managed system (see your

Admin andor Dynatrace API - Authentication)

Mediator creation and configuration

To create a new Dynatrace Managed to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new origin

Press lsquo+rsquo on the SOURCES column to add a new source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoDynatrace Managedrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

37

Now provide the connectivity information for your Dynatrace Managed system

Enter your Dynatrace Managed credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

Dynatrace Managed environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to produce IBM PI CSV files

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

38

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Select the desired Dynatrace Managed metric sources

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

39

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 37: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

37

Now provide the connectivity information for your Dynatrace Managed system

Enter your Dynatrace Managed credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

Dynatrace Managed environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to produce IBM PI CSV files

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

38

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Select the desired Dynatrace Managed metric sources

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

39

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 38: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

38

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Select the desired Dynatrace Managed metric sources

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

39

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 39: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

39

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

lsquoSeparate Files By Metric Typersquo is not used by Dynatrace Managed

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 40: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

40

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 41: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

41

MF NNMi Mediator

Overview

This mediator extracts Metric data from MicroFocusrsquo NNMi system

Extraction Approach

Metric data is extracted from data files produced by the NNMi systemrsquos lsquoCSV Exportrsquo

mechanism - these files are retrieved from the NNMi system via FTP or SFTP

Further info can be found here NNMi CSV export

Pre-requisites

You must have SFTP or FTP credentials to connect to the NNMi system The NNMi system must have been configured to export the required data files The details on this are beyond the scope of this document By default these data files are written by NNMi to varoptOVtmpfinal on the NNMi server eg

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 42: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

42

SWP only retrieves and extracts data from the lsquoRawrsquo files (not Hourly or Daily) For further info on configuring MFNNMi for such periodic data dumping see the relevant documentation or the appropriate Sys Admin

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 43: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

43

Mediator creation and configuration

To create a new MF NNMi to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

Press lsquo+rsquo on the SOURCE column to add a new source

Select lsquoMFNNMirsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your MFNNMi system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 44: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

44

Enter your FTP or SFTP connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the

NNMi environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

Connect to Destination

Now configure the system to be able to connect to the IBM PI REST interface

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 45: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

45

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection for MFNNMi requires

bull Configuration for file handling following data extraction

o Which directory on MF NNMi to look for files varoptOVtmpfinal is the default

o Following file retrieval if lsquoDelete files when donersquo is

Checked - files will be deleted deleted from MF NNMi

Unchecked - files will be movedmoved to the specified lsquoMetric Files Done Directoryrsquo

Note the lsquodonersquo directory does not typically exist and if intended to be used will need to

be manually created on MF NNMi before successful use

bull Selection of Categories (aka lsquoExtensionsrsquo in MF NNMi terminology) of data lsquoComponentrsquo and lsquoInterface

are the only two choices

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 46: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

46

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

You must specify the lsquoPI Topicrsquo you configured (or will configure) on the PI system to receive

the metric data

Note This should be in upper-case per IBM PI requirements

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 47: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

47

Configure Stream

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 48: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

48

Oracle OEM Mediator

Overview

Extraction Approach

Metric data is extracted using JDBC

For more info httpsdocopscacomca-apm10-1enintegratingapi-reference-guidejdbc-api

Oracle OEM Port Defaults

JDBC 1521

Mediator creation and configuration

Select Source

Provide a name for the new source instance

And press lsquoCONTINUErsquo (on lower-right of window)

Connect to Source

Now provide the connectivity information for your Oracle OEM source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 49: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

49

Note The supplied Host Port and Database Name are used in the construction of the JDBC url

jdbcoraclethin$lthostgtltportgtltdatabaseNamegt

Select Destination

Now select the destination

Give this destination a unique name This name will also be used as a prefix for the Oracle OEM

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 50: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

50

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from Oracle OEM is based upon selecting from the available Target Types All

metrics for the selected Target Types within the configured Data Time Window will be

extracted

Select the extraction schedule (Schedule Name)

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 51: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

51

Configuring Distributors

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Save Stream

Then lsquoSAVE STREAMrsquo (or sometimes lsquoSAVE AND START STREAMrsquo)

You should return to Weaver screen and see the stream rendered as below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 52: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

52

Pivotal Cloud Foundry Mediator

Overview

This mediator extracts Metric data from Pivotal Cloud Foundry system

Extraction Approach

Metric data is extracted by tapping in to PCFrsquos lsquofirehosersquo mechanism using a lsquonozzlersquo The

nozzle an executable program when run registers for receipt of PCF metrics and as those

metrics are delivered to the nozzle by PCF they are then pushed to SWPrsquos internal Kafka

system

For details on Pivotal Cloud Foundry lsquoNozzlersquo see Logs Metrics and Nozzles

Pre-requisites

For basic PCF connectivity (used to obtain metadata eg info on applications spaces organizations) you must obtain (probably from your PCF admin)

bull UsernamePassword

bull Email User namePassword

bull PCF hostport info

These will be entered in the SWP Mediator configuration screen Additionally to enable the Nozzle to connect to the PCF firehose and subscribe to metrics yoursquoll also need

bull lsquoUAArsquo (see Pivotal Admin) host

bull usernamepassword

These will be supplied to the nozzle program

Kafka

PCF

Collector

Distributors Nozzl

e

Metricss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 53: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

53

Overall Setup

Configuration for PCF metric collection involves two main steps

bull Configure an SWP Mediation Collector (which is broadly similar to the Mediator configuration for the

other mediators above)

bull Configure and start the Nozzle

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 54: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

54

Mediator creation and configuration

To create a new PivotalCF to PI REST stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new Source

Press lsquo+rsquo on the SOURCES column to add a new source and select lsquoPivotalCFrsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your PivotalCF system

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 55: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

55

Enter your PCF credentials and connection details

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the PCF

environment)

Select Destination

Now select the destination in this case we select IBMPI (REST) though IBMPI (CSV) could be

chosen instead

Press lsquoCONTINUErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 56: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

56

Connect to Destination

Now configure the system to produce IBM PI CSV files

Select Data Types

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 57: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

57

Metric selection from PCF requires setting the following

bull Data extraction schedule

bull Selection of PCF lsquospacesrsquo from which to collect data - keeping in mind the following

o PCF has two kinds of metrics which are

lsquoContainerrsquo metrics - which are metrics for the Containers running on PCF - the

containers lsquocontainrsquo applications and so these are lsquoApplication lsquometrics

Metrics related to the PCF platform itself - these are called lsquoPlatformrsquo metrics

o All lsquoPlatformrsquo metrics are collected

o The lsquoSpacesrsquo selection only affects selection of lsquoApplicationrsquo metrics In PCF Application belong

to Spaces and only metrics for Applications within the selected Spaces will be collected

bull Kafka settings

o Kafka brokers - used by collector to connect to Kafka and retrieve metrics - The default setting is

for the internal SWP Kafka and generally doesnrsquot need to be changed It is exposed to support the

use of an external Kafka system if desired

o The Application and Container Metric Topics are the names of the kafka topics which the

Collector looks to retrieve the two types of metrics The default settings match those of of the

supplied Nozzle configuration and should need to be changed

o Replay mode - this is a test mode which can be used once some data has been delivered to Kafka

by the nozzle It will cause the Collector to continually extract data from the beginning of the

specified Kafka topics This can be useful in testing but in general the default (unchecked) should

left as is

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 58: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

58

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 59: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

59

For PCF itrsquos generally advisable to check the lsquoSeparate Files By Metric Typersquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

Press lsquoSAVE STREAMrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 60: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

60

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 61: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

61

Nozzle Setup

The nozzle executable and associated configuration file can be found in

optIBMstreamweaver-ibmpicontent-packspivotalcfkafka

There are 3 nozzle executables shipped - one for OSX Windows and LinuxCentOS

For the typical IBM PI deployment the LinuxCentOS one is used namely

kafka-firehose-nozzlelinuxamd64

The associated configuration file is

kafka-nozzletoml

The nozzle can be run on the PCF system or co-located with SWP andor on the same host as IBM PI

Copy the kafka-nozzletoml file to a working directory and edit it providing the values shown (these were mentioned in the pre-reqs) Save the file

(Usually something like wssdopplerlthostgt443)

(Usually something like httpsuaahost)

Save the file

Operation

Once the nozzle configuration file has been prepared and streamweaver service has been

started then the nozzle itself can be started

kafka-firehose-nozzlelinuxamd64 -config kafka-nozzletoml

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 62: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

62

Starting the nozzle will cause it to register with the PCF firehose and subscribe lsquolistenrsquo

for metrics Selected metrics will then be delivered to the Kafka instance running within

SWP

Once running you should see output to the nozzle console similar to the following

Note The nozzle is a self-contained executableprocess - you can arrange to run

this from the command line directly as described or as a lsquoservicersquo or otherwise - that

service setup is beyond the scope of this document

To process those PCF metrics and convert them to PI CSV files or deliver them to the

REST if the end-to-end SWP mediator stream must be startedrun in the usual way

To stop processing of PCF metrics

Stop the Mediator stream

Stop the Nozzle (Ctrl-C or otherwise kill the running process)

Note PCF does not persist metrics As metrics are gathered in real-time they are

delivered to via nozzles to any subscribers If the nozzles are not running to collect the

data the data is simply discarded (by PCF) Therefore minimally you should keep the

nozzle and the streamweaver-ibmpi service running to maximize your collection of data

The swpkafka is not intended for long term storage of collected data in the same manner

as a lsquoPerformance Management Systemrsquo but it does give some limited ability to buffer

the metric data (depending on Kafka topic retention configuration)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 63: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

63

SAP Hana Mediator

Overview

This mediator extracts Metric data from SAP Hana database for the user selected tables

Extraction Approach

Metric data is extracted using a JDBC interface Further info can be found here SAP Hana JDBC

SAP Hana Port Defaults

JDBC 30015

Pre-requisites

This integration uses the SAP Hana JDBC driver which is NOT shipped with SWP (due to licensing restrictions) The JDBC driver must be obtained and installed separately

JDBC client can be downloaded from here JDBC Driver page The driver is you should select is the Linux one At time of writing this is in the gzip file hanaclient-23106-linux-x64targz Then

tar xfz hanaclient-23106-linux-x64targz

cd clientclient

tar xfz JDBCTGZ

The driver file is ngdbcjar

Place it in the SWP library directory at optIBMopstream-ibmpilib eg cp ltpathToJDBCDrivergtngdbcjar optIBMopstream-ibmpilib

SWPMediators will require a restart to load this driver

Mediator creation and configuration

To create a new SAP Hana Mediator to PI CSV stream navigate to the SWP WEAVER panel at

httpltmediationservergt7457

Create new source

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 64: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

64

From the Weaver screen press lsquo+rsquo on the Sources column to add a new Source

Then click on lsquoCREATE NEW SOURCErsquo and select lsquoSAPHanarsquo

Provide a name or just go with the default

And press lsquoCONTINUErsquo

Connect to Source

Now provide the connectivity information for your SAP HANA DB system

Enter the connection information for the SAP HANA database

Then press lsquoSAVE ORIGIN AND VALIDATErsquo If you successfully connected you can

continue If the connection check failed you will be alerted and you should make whatever

corrections necessary to the configuration (typically credential issues or reachability to the SAP

HANA host)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 65: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

65

Select Destination

Now select the destination in this case only IBM PI (CSV) is offered as an option You should

almost always select lsquoCREATE NEW DESTINATIONrsquo as a single file stream emerges from

each Destination and we donrsquot generally want to mix data from different Collector instances

Give this destination a unique name This name will also be used as a prefix for the saphana

CSV files sent to PI

Press lsquoCONTINUErsquo

Connect to Destination

Select Data Types

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 66: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

66

Note the items in the right column (Collectors amp Distributors) will not appear until you have

check the lsquoMetricrsquo box

Configuring Collectors

Metric selection from SAP HANA is based upon selecting metric tables and SWP will extract

all metrics for those selected metric tables

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 67: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

67

Select the Metrics Schedule This controls how often extraction attempts are made and is

typically set to the same time as the downstream Predictive Insights aggregation interval

Select the tables you want to extract metric data for (just use the mouse pointer to select as many

hosts as you want)

Specify the Data Window and Data Latency as desired

Then press lsquoCREATE COLLECTORS AND CONTINUErsquo

Configure Distributors

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 68: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

68

Idle File Roll Interval controls how long to wait if a file is idle (no new data being written)

before closing it Files will also close and lsquorollrsquo (be renamed to PI standard CSV file names)

when data for the next time interval arrives

Press lsquoCREATE DISTRIBUTORS AND CONTINUErsquo

Configure Stream

Then name the stream if you want (or just accept the name provided which is assembled from

Origin and Destination names)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 69: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

69

Press lsquoSAVE STREAMrsquo

You should return to WEAVER and see the stream rendered similar to below

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 70: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

70

Operations

Starting metric extraction

Select the collector (or distributor) of the stream you want to start T

Note Just using AppDynamics as an example Starting and stopping streams is the same for all mediators

Then slide the slider to the right to start metric collectionextraction

Stopping metric extraction

To stop extraction move the slider to the left

Running single extractions

Often particularly in early stage configuration it is useful to run metric collection just once ( as opposed to running the periodic metric extraction) To do this navigate to the lsquoDASHBOARDrsquo by selecting lsquoDASHBOARDrsquo on the left hand panel

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 71: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

71

From there click on the three horizontal line icon (to the left of the stream name) You should then see a popup

Click on lsquoRun Tasksrsquo which should bring you to a screen with some summary statistics on the stream in question From here press lsquoRUN ONCErsquo to run the extraction job

Note Before you can access this lsquoRUN ONCErsquo mechanism you must have started the

stream at least once It is ok to turn it off again and lsquoRUN ONCErsquo while the stream is off

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 72: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

72

Editing an existing stream

If you want to revisit and edit the configuration of a particular stream (or component) select the stream and then select the gear icon (to the left of the startstop slider)

You will be presented with panels that let you modify the current stream configuration For example using the AppDynamics stream from above we get

You can navigate through the panels by clicking on the green chevrons (eg the highlighted

green lsquoCOLLECTORrsquo in the above image)

In some cases you may wish to edit the underlying configuration directly in JSON To access the

JSON move the lsquoEdit JSON slider to the right

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 73: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

73

When you have completed your editing press lsquoSAVE amp CLOSErsquo

Note When editing JSON directly you must first press lsquoSAVErsquo on the lower-right of the JSON

edit window then lsquoSAVE amp CLOSErsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 74: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

74

Predictive Insights CSV Output

All metric data extracted by a collector in a stream can be routed to corresponding lsquoPredictive

Insightsrsquo distributor This component will output standard Predictive Insights CSV files in what

is known as lsquoskinnyrsquo or lsquometric-names-in-rowsrsquo format For details on the naming conventions

etc associated with these files see the main Predictive Insights documentation

File details

Files are output to separate directories corresponding to each up-stream distributor

The general file path pattern is as follows

optIBMstreamweaver-ibmpidataswpibmpiIBMPIDestinationltNgtdata

Where N is the ID of the distributor which produces the data

To determine this number go to the WEAVER panel click on the SOURCES-side component

AppD in this example and press the gear icon (to the left of the startstop slider)

Conceptually this reveals the configuration screen for the entire end2end channel

Select the lsquoIBM PREDICTIVE INSIGHTS (CSV) component at the bottom of the screen panel

(you may need to scroll down to see it)

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 75: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

75

Selecting that will reveal the configuration for this component

Edit the JSON (move slider to the right) This reveals the configuration JSON and we can find

the ID of the PI CSV Distributor there - shown as lsquodestinationrsquo

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 76: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

76

In this case the path to data from this distributor is

And examining the directory contents at that path you should see the standard PI csv files (once

you have been runningextracting data)

Timestamps

Timestamps on filenames and in the data itself are in UTC and follow the Java

SimpleDateFormat spec

yyyyMMddHHmmss

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 77: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

77

Predictive Insights lsquomodelrsquo files

A sample Predictive Insights lsquomodelrsquo file is supplied for each of the supported data sources and

these files can be used as a starting point for configuring the Predictive Insights lsquoMediation

Toolrsquo

The model files are located in the directory

optIBMstreamweaver-ibmpicontent-packsibmpicsv

Each one of these files can be loaded into the mediation tool by copying to the Predictive

Insights lsquoMediation Toolrsquo workspace and relevant project eg

cp optIBMstreamweaver-ibmpicontent-

packsibmpicsvpivotalcfpcfpamodel ~workspaceswpdemo

Then when you refresh the view in the Mediation Tool you should see the new pamodel

alongside any other models you may have eg

You must customize the usual items eg File Path for your particular environment

Given that the data is lsquoSkinnyrsquo a single group lsquoallrsquo is defined in each model file with an initial

mapping for each for the supported technologies These are a starting point You will likely

have to change them for your particular deployment context

Group information

If you name your Distributor instance lsquocaapmrsquo or lsquosaphanarsquo for example as created in the

examples above then that will match the group defined in the corresponding pi model file

However if your Distributor instance has a different name then yoursquoll need to create a new

group using the example one as initial guidance This is the preferred approach

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 78: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

78

An alternative is to directly modify the lsquogrouprsquo aspects of the supplied model file by editing the

underlying XML file For example if we wanted to have a different group for appdynamics

vi homescadminworkspaceltmyprojectgtappdynamicspamodel

Locate and change the lsquocaapmrsquo or lsquosaphanarsquo to your preferred group name eg

ltnamegtCAAPMltnamegt

ltgroupsgt

ltnamegtcaapmltnamegt

lttimestampKeygt

Also locate and edit here and similarly change lsquoallrsquo to your new group name

ltsourcesgt

ltnamegtcaapm-caapmltnamegt

ltDataEntitySourcegt

Obtaining metric names for PI Mediation tool

The set of actual metrics is highly dependent on what data is actually extracted from the sources

you will also need to add the metric names The quickest way to do this is to first use the

Mediators to extract representative set of CSV files and from those extract the metric names to a

text file Then use the Predictive Insights Mediation tool to import those metrics For example in

an output file from CA APM you would have CSV files with the following header

TimestampmeasurementValuetargetNameNodemeasu

rementNameunitField1ValueField2ValueField3Val

ueField4ValueField5Value

lsquomeasurementNamersquo column would contain the metrics names we want So we want to process a

set of these output CSV files and extract metricNames from this column

bull CA APM - Column 11

bull SAP HANA - Column 4

The following would achieve this

cd pathtoCSVfiles

grep -v ldquoTimestamprdquo csv | awk -F lsquoprint $11rsquo | sort -u

You can do similar for each of both supported file types adjusting the column number

appropriately to select the desired lsquometricrsquo column

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 79: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

79

Pipe this output to a file to create a file suitable for importing via the lsquoAdd metrics from filersquo

button

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 80: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

80

Predictive Insights REST Output

When configured to deliver data to the IBM PI REST distributor a running channel will POST

data to the IBM PI REST service

No further configuration is necessary

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 81: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

81

Logs

Logs are written to a number of directories under

optIBMstreamweaver-ibmplogs

The main SWP one is swpallinonelog with logs from the other components (eg kafka) also

appearing the same directory

These will often be the first thing that Support will look for when investigating an issue

Changing log levels

Click on lsquoCONFIGURATIONrsquo on the right-hand panel

The right-hand panel allow you to change log levels for various subsystems - left-click on LOG

LEVELS

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 82: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

82

Then you can change the log-levels for individual components

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 83: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

83

Support

To get assistance from the IBM support team you may open a Problem Management Record

(PMR)

The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM

Operations Analytics Advance Insights indicating the ldquoMediation Pack for Solution Namerdquo

Describe the problem encountered providing the following information that must be attached to

the PMR

Product Name IBM Operations Analytics Predictive Insights or IBM Operations Analytics

Advance Insights

Product Version 136 or higher

Mediation Pack Mediation Pack Group 2

Mediation Pack Version 15

Include the following

bull Sample extracted CSV files which are being ingested into Predictive Insights

bull Data source connectivity status screenshot

bull Log files

o Predictive Insights log files ($PI_HOMElogtopicname) when the

problem occurred while using the mediation pack

o Mediation pack log files ( optIBMstreamweaver-ibmpilog)

bull Run the $PI_HOMEbincollectsh and return collected info

bull A screen shot of the error message that is generated while using the mediation

pack

bull A specific use case scenario(s) where problem occurred while using the

mediation pack

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 84: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

84

Appendix 1 SAP HANA metrics

The metrics for SAP HANA are extracted from the tables shown below (For ref httpshelpsapcomviewer4fe29514fd584807ac9f2a04f67547672003en-USd234eedbd29510148efbf332391de7fdhtml) Also fields typically used for resourceattributes are mapped to lsquokeyrsquo fields also as shown below Table HOST_WORKLOAD Metrics COMMIT_COUNT COMPILATION_COUNT CURRENT_COMMIT_RATE CURRENT_COMPILATION_RATE CURRENT_EXECUTION_RATE CURRENT_MEMORY_USAGE_RATE CURRENT_ROLLBACK_RATE CURRENT_TRANSACTION_RATE CURRENT_UPDATE_TRANSACTION_RATE EXECUTION_COUNT PEAK_COMMIT_RATE PEAK_COMPILATION_RATE PEAK_EXECUTION_RATE PEAK_MEMORY_USAGE_RATE PEAK_ROLLBACK_RATE PEAK_TRANSACTION_RATE PEAK_UPDATE_TRANSACTION_RATE ROLLBACK_COUNT UPDATE_TRANSACTION_COUNT Keys HOST -gt Key1 PORT -gt Key2 Table HOST_DATA_VOLUME_PAGE_STATISTICS_BASE Metrics FILL_RATIO INITIAL_BLOCK_COUNT PAGE_SIZE SHADOW_BLOCK_COUNT SUPERBLOCK_COUNT TOTAL_ALLOCATE_BLOCK_COUNT TOTAL_SET_BLOCK_FREE_AFTER_SAVEPOINT_COUNT TOTAL_SET_BLOCK_FREE_COUNT USED_BLOCK_COUNT Keys

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 85: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

85

HOST -gt Key1 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_STATISTICS_BASE Metrics ACTIVE_REQUEST_COUNT ACTIVE_THREAD_COUNT ALL_FINISHED_REQUEST_COUNT AVAILABLE_MEMORY FINISHED_NON_INTERNAL_REQUEST_COUNT OPEN_FILE_COUNT PENDING_REQUEST_COUNT PHYSICAL_MEMORY PROCESS_CPU_TIME PROCESS_MEMORY PROCESS_PHYSICAL_MEMORY REQUESTS_PER_SEC RESPONSE_TIME THREAD_COUNT TOTAL_CPU_TIME TOTAL_MEMORY Keys HOST -gt Key1 PORT -gt Key2 PROCESS_ID -gt Key3 SERVICE_NAME -gt Key4 DETAIL -gt Key5 Table HOST_RESOURCE_UTILIZATION_STATISTICS_BASE Metrics ALLOCATION_LIMIT FREE_PHYSICAL_MEMORY FREE_SWAP_SPACE INSTANCE_CODE_SIZE INSTANCE_SHARED_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_ALLOCATED_SIZE INSTANCE_TOTAL_MEMORY_USED_SIZE TOTAL_CPU_IDLE_TIME TOTAL_CPU_SYSTEM_TIME TOTAL_CPU_USER_TIME TOTAL_CPU_WIO_TIME USED_PHYSICAL_MEMORY USED_SWAP_SPACE Keys HOST -gt Key1 Table HOST_DATA_VOLUME_SUPERBLOCK_STATISTICS_BASE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 86: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

86

Metrics FILL_RATIO SUPERBLOCK_COUNT Keys HOST -gt Key1 PORT -gt Key2 DATA_VOLUME_NAME -gt Key3 Table HOST_SERVICE_MEMORY Metrics ALLOCATION_LIMIT CODE_SIZE COMPACTORS_ALLOCATED_SIZE COMPACTORS_FREEABLE_SIZE EFFECTIVE_ALLOCATION_LIMIT HEAP_MEMORY_ALLOCATED_SIZE HEAP_MEMORY_USED_SIZE LOGICAL_MEMORY_SIZE PHYSICAL_MEMORY_SIZE SHARED_MEMORY_ALLOCATED_SIZE SHARED_MEMORY_USED_SIZE STACK_SIZE TOTAL_MEMORY_USED_SIZE Keys HOST -gt Key1 PORT -gt Key2 SERVICE_NAME -gt Key3 Table HOST_HEAP_ALLOCATORS_BASE Metrics DEPTH EXCLUSIVE_ALLOCATED_COUNT EXCLUSIVE_ALLOCATED_SIZE EXCLUSIVE_ALLOC_ERRORS EXCLUSIVE_COUNT_IN_USE EXCLUSIVE_DEALLOCATED_COUNT EXCLUSIVE_DEALLOCATED_SIZE EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE EXCLUSIVE_PEAK_ALLOCATION_SIZE EXCLUSIVE_SIZE_IN_USE INCLUSIVE_ALLOCATED_COUNT INCLUSIVE_ALLOCATED_SIZE INCLUSIVE_COUNT_IN_USE INCLUSIVE_DEALLOCATED_COUNT INCLUSIVE_DEALLOCATED_SIZE INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE INCLUSIVE_PEAK_ALLOCATION_SIZE

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 87: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

87

INCLUSIVE_SIZE_IN_USE MALLOC_PROXY_CACHE_MISSES Keys HOST -gt Key1 PORT -gt Key2 CATEGORY -gt Key3

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 88: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

88

reg

copy Copyright IBM Corporation 2019 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionall y equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the users responsibility to evaluate and verify the operation of any non-IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to

IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504-1785 USA

For license inquiries regarding double-byte (DBCS) information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623-14 Shimotsuruma Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON-INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE

Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements andor changes in the product(s) andor the program(s) described in this publication at any time without notice Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged should contact

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 89: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

89

IBM Corporation 2Z4A101 11400 Burnet Road Austin TX 78758 USA Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this document should verify the applicable data for their specific environment Information concerning non-IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non-IBM products Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products All statements regarding IBMs future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed

Copyright License This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs The sample programs are provided AS IS without warranty of any kind IBM shall not be liable for any damages arising out of your use of the sample programs

Privacy Policy Considerations IBM Software products including software as service solutions (ldquoSoftware Offeringsrdquo) may use cookies or other technologies to collect product usage information to help improve the end user experience to tailor interactions with the end user or for other purposes In many cases no personally identifiable information is collected by the Software Offerings Some of our Software Offerings can help enable you to collect personally identifiable information If this Software Offering uses cookies to collect personally identifiable information specific information about this offeringrsquos use of cookies is set forth below Depending on the configuration that is deployed in the application server this Software Offering may use session cookies for session management These cookies can optionally be enabled or disabled but disabling the cookies will also disable the functionality that they enable See the application server documentation for additional information Optionally you can also use functionality such as typeahead to store data locally on the browser The data that is stored locally could contain personally identifiable information Data validation If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies you should seek your own legal advice about any laws that are applicable to such data collection including any requirements for notice and consent For more information about the use of various technologies including cookies for these purposes see IBMrsquos Privacy Policy at httpwwwibmcomprivacy and IBMs Online Privacy Statement at httpwwwibmcomprivacydetails in the section entitled ldquoCookies Web Beacons and Other Technologiesrdquo and the IBM Software Products and Software-as-a-Service Privacy Statement at httpwwwibmcomsoftwareinfoproduct-privacy

Trademarks

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics
Page 90: IBM Operations Analytics Predictive Insights › itom › wp-content › uploads › sites › 39 › 2… · ibm operations analytics predictive insights mediation pack group 2 iv

IBM Operations Analytics Predictive Insights Mediation Pack Group 2

90

IBM the IBM logo and ibmcom are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at ldquoCopyright and trademark informationrdquo at wwwibmcomlegalcopytradeshtml J9 Technologies and the J9 Technologies logo are registered trademark andor trademarks of J9 Technologies in various jurisdictions Adobe Acrobat PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle andor its affiliates Linux is a trademark of Linus Torvalds in the United States other countries or both Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries or both UNIX is a registered trademark of The Open Group in the United States and other countries Other product and service names might be trademarks of IBM or other companies

  • Mediation Pack Group 2
  • Contents
  • Overview
    • Prerequisite Information
      • SWP Mediators Overview
        • Key steps to setting up Mediators
          • Installing and Configuring
            • Zip file contents
            • Install procedure
              • New Installation
              • Notes
                • Installing to an alternative location
                • Running as non-`scadmin` user
                • Co-existence of SWP-based and Accelerator-based mediators
                • Data directories
                • Installation users
                  • Starting SWP system
                  • Stopping SWP system
                  • Upgrade installation
                    • Deployed directory contents
                      • optIBMstreamweaver-ibmpi
                      • jre
                      • bin
                      • content-packs
                      • lib
                      • logs
                      • conf
                      • services
                      • dataswp
                        • Accessing the Mediation system
                          • AppDynamics Mediator
                            • Overview
                            • Mediator creation and configuration
                            • Select Source
                            • Connect to Source
                            • Select Destination
                            • Connect to Destination
                            • Select Data Types
                            • Configuring Collectors
                            • Configuring Distributors
                            • Save Stream
                              • AWS Cloudwatch Mediator
                                • Overview
                                • Extraction Approach
                                  • Pre-requisites
                                    • Mediator creation and configuration
                                    • Create new Source
                                    • Connect to Source
                                    • Select Destination
                                    • Connect to Destination
                                    • Select Data Types
                                    • Configuring Collectors
                                    • Configure Distributors
                                    • Configure Stream
                                      • CA APM (Introscope) Mediator
                                        • Overview
                                        • Note
                                        • Mediator creation and configuration
                                        • Select Source
                                        • Connect to Source
                                        • Select Destination
                                        • Connect to Destination
                                        • Select Data Types
                                        • Configuring Collectors
                                        • Configuring Distributors
                                        • Configure Stream
                                          • Dynatrace Managed
                                            • Overview
                                            • Extraction Approach
                                              • Pre-requisites
                                                • Mediator creation and configuration
                                                • Create new origin
                                                • Connect to Source
                                                • Select Destination
                                                • Connect to Destination
                                                • Select Data Types
                                                • Configuring Collectors
                                                • Configure Distributors
                                                • Configure Stream
                                                  • MF NNMi Mediator
                                                    • Overview
                                                    • Extraction Approach
                                                      • Pre-requisites
                                                        • Mediator creation and configuration
                                                        • Create new source
                                                        • Connect to Source
                                                        • Select Destination
                                                        • Connect to Destination
                                                        • Select Data Types
                                                        • Configuring Collectors
                                                        • Configure Distributors
                                                        • Configure Stream
                                                          • Oracle OEM Mediator
                                                            • Overview
                                                            • Mediator creation and configuration
                                                            • Select Source
                                                            • Connect to Source
                                                            • Select Destination
                                                            • Connect to Destination
                                                            • Select Data Types
                                                            • Configuring Collectors
                                                            • Configuring Distributors
                                                            • Save Stream
                                                              • Pivotal Cloud Foundry Mediator
                                                                • Overview
                                                                • Extraction Approach
                                                                  • Pre-requisites
                                                                    • Overall Setup
                                                                    • Mediator creation and configuration
                                                                    • Create new Source
                                                                    • Connect to Source
                                                                    • Select Destination
                                                                    • Connect to Destination
                                                                    • Select Data Types
                                                                    • Configuring Collectors
                                                                    • Configure Distributors
                                                                    • Configure Stream
                                                                    • Nozzle Setup
                                                                    • Operation
                                                                      • SAP Hana Mediator
                                                                        • Overview
                                                                        • Extraction Approach
                                                                          • SAP Hana Port Defaults
                                                                          • Pre-requisites
                                                                            • Mediator creation and configuration
                                                                            • Create new source
                                                                            • Connect to Source
                                                                            • Select Destination
                                                                            • Connect to Destination
                                                                            • Select Data Types
                                                                            • Configuring Collectors
                                                                            • Configure Distributors
                                                                            • Configure Stream
                                                                              • Operations
                                                                                • Starting metric extraction
                                                                                • Stopping metric extraction
                                                                                • Running single extractions
                                                                                • Editing an existing stream
                                                                                  • Predictive Insights CSV Output
                                                                                    • File details
                                                                                    • Timestamps
                                                                                      • Predictive Insights lsquomodelrsquo files
                                                                                        • Group information
                                                                                        • Obtaining metric names for PI Mediation tool
                                                                                          • Predictive Insights REST Output
                                                                                            • Logs
                                                                                              • Changing log levels
                                                                                                  • Support
                                                                                                  • Appendix 1 SAP HANA metrics

Recommended