Post on 20-May-2020
transcript
Operations Analytics Predictive Insights
IBM Operations Analytics Predictive Insights Mediation Pack for Azure
Version 1.1.0
Operations Analytics Predictive Insights Azure Mediation Pack
2
Edition notice
This edition applies to IBM Operations Analytics Predictive Insights version 1.3.3 and to all subsequent releases.
Please refer to the IBM developerWorks terms of use for more information. Also reference the IBM Operations
Analytics Development Community (https://developer.ibm.com/itoa/) for additional information on Predictive
Insights.
Operations Analytics Predictive Insights Azure Mediation Pack
3
Contents Overview ................................................................................................................................................ 3
Prerequisite Information ........................................................................................................................ 4
Azure Performance Metrics .................................................................................................................... 5
Mediation Packs Contents ................................................................................................................... 7
Configure Data Modeling ................................................................................................................... 10
Create Topic ...................................................................................................................................... 13
Deploy Model .................................................................................................................................... 14
Start Training ..................................................................................................................................... 14
Review Training Process .................................................................................................................... 15
Review Anomalies ............................................................................................................................. 15
Support ................................................................................................................................................. 18
Notices .................................................................................................................................................. 19
Overview IBM Operations Analytics Predictive Insights analyzes data and learns the normal behavior of a system
through KPI metric data streams. This includes data from various sources including metrics gathered by
Operations Analytics Predictive Insights Azure Mediation Pack
4
industry performance and network monitoring solutions. A mediation pack helps users quickly integrate
the KPI data streams by providing a default model to integrate a performance manager. IBM Operations
Analytics Predictive Insights Mediation Pack for Azure provides a pre-built model for data ingestion.
Predictive Insights creates a performance model that it uses to detect or forecast behavior outside the
modeled range, including generating an alarm when an anomalous behavior occurs. Doing so enables
organizations to pro-actively avoid system and solution outages before users are impacted. The
Mediation Pack for Azure provides the capability to ingest performance data and metrics from Azure
into IBM Operations Analytics Predictive Insights to perform analysis. This document will cover the steps
to setup the Mediation Pack and adjust the data stream for deployment in a customer environment.
Azure Mediation Pack Architecture (CSV)
Mediation Pack development considers all possible integration options to determine the optimal
configuration. Options are reviewed based on the source systems ability to present data using multiple
methods. The selected methods are derived based on many factors including but not limited to
flexibility, scalability, and availability of KPI’s (Key Performance Indicators). Additional considerations are
based on the source systems majority configuration and maturity of such methods at the time of
development. Requests to enhance mediation packs utilizing additional methods supported by the
source system can be reviewed with our professional services team.
Prerequisite Information
Operations Analytics Predictive Insights Azure Mediation Pack
5
Item Description Name Mediation Pack for Azure
Java IBM Java 8 / oracle 1.8
OS Linux
Predictive Insights supported Versions V1.3.3, V1.3.5 and 1.3.6.
Predictive Insights Information IBM Operations Analytics (demo, trial, resources)
https://www.ibm.com/us-en/marketplace/it-operations-analytics
IBM Operations Analytics Community (docs, videos, mediation packs) https://developer.ibm.com/itoa/
IBM Operations Analytics Predictive Insights documentation
http://www.ibm.com/support/knowledgecenter/SSJQQ3
Filename Azure_MedPack_v1.1.0.zip
Data source Every 5 minutes, the run.sh script in this pack extracts the
csv files which later collected by Predictive Analytics.
• Azure needs to be configured for historical
Additional Information collection for metrics and collecting performance metric data for every 5 min
Azure Performance Metrics
List of Azure resource types that supports metrics along with the metric namespaces given in the
<install_Dir>/Azure_MedPack_v1.1.0/azure_metrics directory.
Here all_azure_metrics_05_20_2019.txt contains list of all available resource types along with
corresponding metric names.recommended_azure_metrics_05_20_2019.txt contains the metrics that
are recommended for azure with respect to the Operations Analytics Predictive Insights.
This pack was tested with resource type Microsoft.Compute/virtualMachines and corresponding
metrics.
Installing and Configuring
Get Azure Auth information for configuring pack, using below steps.
Operations Analytics Predictive Insights Azure Mediation Pack
6
(Ref: https://docs.microsoft.com/en-us/rest/api/azure/)
Azure Cloud Shell
Go to Azure Cloud Shell
Local with Azure CLI
Install Azure CLI 2.0
Login
az login
Set Active Subscription
az account set --subscription "your subscription name or id
Create Service Principal
az ad sp create-for-rbac -n "your service principal name"
Copy this output to a temp location, you will need the values for Pack configuration
Service Principal Password Reset
You can execute the following command if you ever need to reset your Service Principal password.
az ad sp reset-credentials --name "your service principal name"
IBM JAVA (1.8 and above)
Make Sure Java installed on the machine where the pack is going to install. Verify the java version using
below command. Should give the out put of JRE version running on the machine.
Java -version
Operations Analytics Predictive Insights Azure Mediation Pack
7
Mediation Packs Contents The following are contained in the Mediation Pack: Azure_MedPack_v1.1.0.zip
•all_azure_metrics_05_20_2019.txt
•recommended_azure_metrics_05_20_2019.txt
azure_metrics
•run.sh
/bin
•Azure_MediationPack_UserGuide_v1.1.0.pdf
•Azure_MediationPack_README_v1.1.0.txt
/docs
•authBean.class
•AzureMain.class
•getAuthToken.class
•getMetricData.class
•ListResource.class
•ListResourceBean.class
•MainObject.class
•Name.class
•ResourceValueBean.class
•Timeseries.class
•TimeseriesData.class
•tsToSec8601.class
•Value.class
classes
•application.properties
conf
•run.sh
bin
•Azure_MedPack.log
logs
•com.fasterxml.jackson.annotations.jar
•com.fasterxml.jackson.core.jar
•com.fasterxml.jackson.databind.jar
lib
etc
•azure.pamodel
/pamodel
•virtualMachines_2019_08_21_20_50_00.csv
/sample
license
Operations Analytics Predictive Insights Azure Mediation Pack
8
/bin/run.sh - for run the mediation pack to collect the performance metrics.
/class - Directory contains the binary files needed by the Mediation Pack.
/conf - Directory contains the config file for the pack.
/lib - Directory contains the jar files
/logs/Azure_MedPack.log - Directory contains the logfile of the Mediation Pack.
/docs - Users Guide for Mediation Pack of Azure for deploying and configuring the Mediation Pack.
Azure_MediationPack_UserGuide_v1.1.0.pdf, Azure_MediationPack_README_v1.1.0
This is a deployment procedural guide to deploy the Azure mediation pack into the Predictive
Insights environment.
/license - Text license files for governing the use of the Mediation pack.
/pamodel - Core files used in the Mediation Pack representing the Predictive Insights model consisting of
all attributes and metric definitions.
azure.pamodel
The azure.pamodel file is a Predictive Insights model file which consists of all attributes and metric
definitions. This file consists of all metric group definitions. Import this file into mediation tool.
/sample
Contains a sample CSV file representing the captured data streams that can be used to try the Mediation
Pack output in the Operations Analytics Predictive Insights product. There will be one csv file for each
resource type.
Operations Analytics Predictive Insights Azure Mediation Pack
9
Mediation Pack property file configuration
tokenUrl: Which is equal to https://login.microsoftonline.com/{tenantId}/oauth2/token
{tenantId} id replace with the value with above genrated tenantId.
(look into the step "Create Service Principal" in the output token is the tenantId)
grant_type: Which is equal to "client_credentials"
client_id: Which is equal to the appId genrated in the step "Create Service Principal"
client_secret: Which is equal to the password genrated in the step "Create Service Principal"
resource: Which is equal to "https://management.azure.com"
subscription_id: subscription id by running the following command
az account show --query id
csv_dir: Which is the directory where the generated csv files will save.
resource_types: List of resources types for which the metric information needed with (,) separated.
After resource_types, Give each resource type with required metrics as given in the default
application.properties file.
List of All azure metrics and recommended metrics were given in the
<install_Dir>/Azure_MedPack_1.1.0/azure_metrics directory. Copy the line for which resource type
required and past into the property file after resource_type property .user may not need to provide all
the metrics of resource type but the one requires to monitor. Like the list of recommended metrics that
are provided above.
timespan: The length of the interval to query. 5 minutes is the default in the application.properties file.
Latency: This is the difference between the current time to the timespan. It is better to gather the
metrics for few minutes past rather than last 5 minutes from the current time in order to be confident
that all the data for that interval is available in Azure
Configuring Mediation Pack in Cron
Provide execute permission to run.sh using below command
chmod 775 <install_Dir>Azure_MedPack_1.1.0/bin/run.sh
To collect the data for every 5 minutes interval. run.sh script should be run in cron.
Below is the sample cron expression to run for every five minutes.
*/5 * * * * <install_Dir>Azure_MedPack_1.1.0/bin/run.sh
Operations Analytics Predictive Insights Azure Mediation Pack
10
Configure Data Modeling
Follow the below procedure to configure the azure model
1. Copy the azure.pamodel file to the Predictive Insights server where the mediation tool
installed.
2. Launch the mediation tool using $PI_HOME/bin/mediationtool.sh
3. Go to File and click on import button
Note: The model is pre-defined with some selected metrics. Make sure the mediation model is updated with all metrics
and/or metric groups that are present in pack_filetered_metrics.py file before deployment. Refer ‘Configuration and
Administration Guide’ of IBM Operations Analytics Predictive Insights for model configuration (chapter 2).
Operations Analytics Predictive Insights Azure Mediation Pack
11
4. Select the Predictive Insights Model in the menu and click on Next
5. Select the azure.pamodel file and click on ok
Operations Analytics Predictive Insights Azure Mediation Pack
12
6. Click on finish to complete the import process
7. Enter the azure Predictive Insights Data source connection details.
For more information refer to the following url:
https://www.ibm.com/support/knowledgecenter/SSJQQ3_1.3.6/com.ibm.scapi.doc/admin_gui
de/t_tasp_admin_createfilesystemdatasource.html
10. Change to the model design tab you will able to see all metric groups
11. Select each metric group in turn and preview the data by clicking the preview button near the
top-right.
Operations Analytics Predictive Insights Azure Mediation Pack
13
Create Topic
Follow below procedure to create topic in Predictive Insights.
1. Login into the Predictive Insights server as scadmin
2. Change to $PI_HOME/bin
3. Create the topic using ./admin.sh create_topic azure
4. ./admin.sh set -t=azure system.aggregation.interval 5
For more information on properties for the admin.sh refer to the following:
http://www.ibm.com/support/knowledgecenter/SSJQQ3_1.3.6/com.ibm.scapi.doc/adm
in_guide/r_tsaa_adminguide_cli_show_instance_properties.html
Refer to the following for the set parameter:
http://www.ibm.com/support/knowledgecenter/SSJQQ3_1.3.6/com.ibm.scapi.doc/adm
in_guide/r_tsaa_adminguide_cli_set_property.html
Refer to the following on the properties that can be set:
http://www.ibm.com/support/knowledgecenter/SSJQQ3_1.3.6/com.ibm.scapi.doc/adm
in_guide/c_tsaa_adminguide_config_properties.html
Operations Analytics Predictive Insights Azure Mediation Pack
14
Deploy Model
1. Click on the model(s) to be deployed and multi-select if multiple models are to be deployed to
the same topic. Then Right click on the selection and select Deploy Model option.
2. Provide the Predictive Insights hostname, port number and database name
3. Provide the password for scadmin and click on ok
4. Click on Yes to deploy the model
Start Training 1. Login into the Predictive Insights server as scadmin
2. Change to $PI_HOME/bin
3. Run ./start.sh -t=azure to start the topic
4. Start the training using below command. -s start time is based on your datasource collection.
-l is the time it takes for an interval to have fully arrived from the datasource.
For example ./admin.sh run_extractor_instance -t=azure -s=20170917-1610 -l=60
Operations Analytics Predictive Insights Azure Mediation Pack
15
Review Training Process
Once training has started you will see events in the Active event list
1. Login in to https://hostname:16311/ibm/console
2. User name: ncoadmin
3. Enter ncoadmin password
4. You can see the list events showing status about training process in Active event list
Review Anomalies 1. Login in to https://hostname:16311/ibm/console
2. User name: ncoadmin
3. Enter ncoadmin password
4. Navigate to the Detected Anomalies Page
Operations Analytics Predictive Insights Azure Mediation Pack
16
5. Review the Anomalies
6. Right click on an anomaly and select Service Diagnosis tool
Operations Analytics Predictive Insights Azure Mediation Pack
17
7. Review the affected metrics and behavior of the metrics for the period
Operations Analytics Predictive Insights Azure Mediation Pack
18
Support
To get assistance from the IBM support team, you may open a Problem Management Record (PMR).
The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM Operations
Analytics Advanced Insights indicating the "Mediation Pack for Solution Name". Describe the problem
encountered providing the following information that must be attached to the PMR:
Product Name: IBM Operations Analytics Predictive Insights or IBM Operations Analytics Advanced
Insights
Product Version: 1.3.3 or higher.
Mediation Pack: Mediation Pack for Azure
Mediation Pack Version: 1.1.0
Include the following:
1. Sample extracted CSV files, which are being ingested into Predictive Insights.
Attach the output of the command $PI_HOME/bin/collect.sh -t=<topic>
2. Data source connectivity status screenshot
3. Predictive Insights log files ($PI_HOME/log/topicname/) when the problem occurred while
using the mediation pack.
4. A screen shot of the error message that is generated while using the mediation pack.
5. A specific use case scenario(s) where problem occurred while using the mediation pack.
6. Azure_MedPack.log file after running the pack. Which will be in <Install_Dir>/logs/ directory.
Operations Analytics Predictive Insights Azure Mediation Pack
19
Notices
© Copyright IBM Corporation 2019
IBM United States of America
Produced in the United States of America
US Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for
information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to
state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does
not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation
of any non-IBM product, program, or service.
IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does
not give you any license to these patents. You can send license inquiries, in writing, to:
IBM Director of Licensing IBM Corporation
North Castle Drive Armonk, NY 10504-1785 U.S.A.
For license inquiries regarding double-byte (DBCS) information, contact the IBM Intellectual Property Department in your country or send
inquiries, in writing, to:
Intellectual Property Licensing
Legal and Intellectual Property Law
IBM Japan, Ltd.
1623-14, Shimotsuruma, Yamato-shi
Kanagawa 242-8502 Japan
The following paragraph does not apply to the United Kingdom or any other country where such provisions are
inconsistent with local law:
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you.
This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these
changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the
program(s) described in this publication at any time without notice.
Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of
those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk.
IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you.
Licensees of this program who wish to have information about it for the purpose of enabling: (i) the exchange of information between
independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged,
should contact:
Operations Analytics Predictive Insights Azure Mediation Pack
20
IBM Corporation 2Z4A/101
11400 Burnet Road Austin, TX 78758 U.S.A.
Such information may be available, subject to appropriate terms and conditions, including in some cases payment of a fee.
The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer
Agreement, IBM International Program License Agreement or any equivalent agreement between us.
Any performance data contained herein was determined in a controlled environment. Therefore, the results obtained in other operating
environments may vary significantly. Some measurements may have been made on development-level systems and there is no guarantee that
these measurements will be the same on generally available systems. Furthermore, some measurement may have been estimated through
extrapolation. Actual results may vary. Users of this document should verify the applicable data for their specific environment.
Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly
available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related
to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.
All statements regarding IBM's future direction or intent are subject to change or withdrawal without notice, and represent goals and
objectives only.
This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the
examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names
and addresses used by an actual business enterprise is entirely coincidental.
If you are viewing this information in softcopy form, the photographs and color illustrations might not be displayed.
Copyright License
This information contains sample application programs in source language, which illustrate programming techniques on various operating
platforms. You may copy modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing,
using, marketing or distributing application programs conforming to the application programming interface for the operating platform for
which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot
guarantee or imply reliability, serviceability, or function of these programs. The sample programs are provided "AS IS", without warranty of any
kind. IBM shall not be liable for any damages arising out of your use of the sample programs.
Privacy Policy Considerations
IBM Software products, including software as service solutions, (“Software Offerings”) may use cookies or other technologies to collect product
usage information, to help improve the end user experience, to tailor interactions with the end user, or for other purposes. In many cases, no
personally identifiable information is collected by the Software Offerings. Some of our Software Offerings can help enable you to collect
personally identifiable information. If this Software Offering uses cookies to collect personally identifiable information, specific information
about this offering’s use of cookies is set forth below.
Depending on the configuration that is deployed in the application server, this Software Offering may use session cookies for session
management. These cookies can optionally be enabled or disabled, but disabling the cookies will also disable the functionality that they enable.
See the application server documentation for additional information.
Optionally, you can also use functionality, such as typeahead, to store data locally on the browser. The data that is stored locally could contain
personally identifiable information: Data validation.
If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable
information from end users by using cookies and other technologies, you should seek your own legal advice about any laws that are applicable
to such data collection, including any requirements for notice and consent.
For more information about the use of various technologies, including cookies, for these purposes, see IBM’s Privacy Policy
athttp://www.ibm.com/privacy and IBM's Online Privacy Statement at http://www.ibm.com/privacy/details in the section entitled “Cookies,
Operations Analytics Predictive Insights Azure Mediation Pack
21
Web Beacons and Other Technologies” and the "IBM Software Products and Software-as-a-Service Privacy Statement"
at http://www.ibm.com/software/info/product-privacy.
Trademarks
IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corp., registered in many
jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies.
A current list of IBM trademarks is available on the Web at “Copyright and trademark information” at www.ibm.com/legal/copytrade.shtml.
Adobe, Acrobat, PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in
the United States, other countries, or both.
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates. Linux is a trademark of
Linus Torvalds in the United States, other countries, or both.
Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or
both.
UNIX is a registered trademark of The Open Group in the United States and other countries. Other product and service names might be
trademarks of IBM or other companies.