User ManualCG Elastic
Version 1.17-SNAPSHOT
2017-11-30
Table of Contents1. Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
2. Modules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.1. HeartBeat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.2. MetricBeat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.3. CgWmBeat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.3.1. ISHome . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.3.2. ISService . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.3.3. UmQueue. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.3.4. UmDoc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.3.5. Tablespace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.3.6. Perflog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.3.7. Sysmem, Syscpu, Sysdisk (deprecated) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.4. Jenkins. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Table 1. History
Date Author Detail
2017-10-09 bcouetil Minor changes
2017-10-06 bcouetil CgElastic beat migration + CgUtils DB update
2017-09-15 bcouetil Anomaly code refactoring
1. IntroductionwebMethods offer some nice KPI and dashboard handling by the hand of Optimize For Process,Optimize For Infra, and MashZone.
The setup, from which CgElastic is a part of, involves the (free and versatile) Elastic Stack instead(Elasticsearch, Logstash, Beats, Kibana).
Figure 1. webMethods Architecture involving Elastic Stack
1
Figure 2. Kibana dashboard on a webMethods project
Figure 3. Graphana dashboards on a webMethods project
2
2. ModulesIn the following paragraphs, different modules are presented. All have the purpose to populate anElastisearch. Some are official modules applied to webMethods ecosystem.
2.1. HeartBeatThe official Elastic module HeartBeat is able to ping a host regularly through various channels. Itcan be used to check that the IS, UM, MWS, CentraSite modules are up and running, as wells ascontinuous deployment tools such as Gerrit, Jenkins, SonarQube, Nexus.
Figure 4. Number of healthchecks KO checked every 30s
Example JSON Data
{ "_index": "heartbeat-2017.09.28", "_type": "doc", "_id": "AV7IZ1_jh9aXLVWW6fIY", "_version": 1, "_score": null, "_source": { "@timestamp": "2017-09-28T12:12:50.727Z", "beat": { "hostname": "2008c14c7d19", "name": "2008c14c7d19", "version": "5.5.2" }, "duration": { "us": 22723 }, "host": "frpardge", "response": { "status": 200 }, "type": "Kibana", "up": true, "url": "http://frpardge:5601/app/kibana#/management?_g=%28%29" }}
3
2.2. MetricBeatThe official Elastic module MetricBeat gives insights on Memory, CPU, disk, network of the currenthost. It is able to give these insight on Docker server and containers as well.
Figure 5. CPU usage overtime
Figure 6. Memory usage overtime
Figure 7. Docker CPU usage overtime
4
Figure 8. Docker memory usage overtime
Example JSON Data
{ "_index": "metricbeat-2017.09.28", "_type": "doc", "_id": "AV7IZkNlh9aXLVWW6e3m", "_version": 1, "_score": null, "_source": { "@timestamp": "2017-09-28T12:11:37.813Z", "beat": { "hostname": "frpardge", "name": "frpardge", "version": "5.5.2" }, "docker": { "info": { "containers": { "paused": 0, "running": 6, "stopped": 3, "total": 9 }, "id": "KRQF:O4Q2:C3XT:HVLP:LKJW:RO5A:XB55:BKTR:K5RB:LK5R:63EJ:6BPB", "images": 82 } }, "metricset": { "host": "/var/run/docker.sock", "module": "docker", "name": "info", "rtt": 39687 }, "type": "metricsets" }}
5
2.3. CgWmBeatCapgemini webMethods beat module cgwmbeat gathers all custom statistics from webMethodsservers. There are several area of data detailed in next paragraphs.
2.3.1. ISHome
The ISHome module gives insights on Integration Server overall statistics
Figure 9. IS Memory usage
Figure 10. IS uptime
6
Example JSON Data
{ "_index": "isstat", "_type": "isstat", "_id": "AV7IaOwuh9aXLVWW6fiJ", "_version": 1, "_score": null, "_source": { "startTime": "2017-09-28 12:36:40 CEST", "uptime": "1h:37m:52s", "maxMem": 932352, "totalMem": 631808, "freeMem": 364499, "usedMem": 267309, "freeMemPer": 57, "usedMemPer": 43, "memArrayMax": 801792, "svrT": 2, "svrTMax": 14, "srvThreadCount": 1, "srvThreadCountPeak": 11, "sysT": 374, "sysTMax": 375, "conn": 55, "connMax": 55, "connAvg": 322, "reqTotal": 31, "reqLifetime": 3714, "reqAvg": 165, "reqTotAvg": 209, "newReqPM": 36, "endReqPM": 36, "errSvc": 104, "errSys": 43, "svcRate": 100, "ssnUsed": 3, "ssnPeak": 4, "ssnMax": 2147483647, "connStatefulSessionsCurrent": 0, "connStatefulSessionsPeak": 0, "connStatefulSessionsLimit": 2147483647, "insertDate": "2017-09-28T12:14:33.005" }, "fields": { "insertDate": [ 1506600873005 ], "usedMemPretty": [ 267309000 ], "totalMemPretty": [ 631808000 ], "maxMemPretty": [ 932352000 ] }, "sort": [ 1506600873005 ]}
7
2.3.2. ISService
The ISService module gathers webMethods services running statistics.
Figure 11. IS Services total access
Example JSON Data
{ "_index": "servicestat", "_type": "startup", "_id": "AV7TWV-fh9aXLVWWGoGl", "_version": 1, "_score": null, "_source": { "fullName": "wm.xslt.Admin:startup", "package": "wm.xslt.Admin", "shortName": "startup", "accessLastDate": "2017-09-28 20:31:46 CEST", "accessTotal": 1, "accessNew": 0, "accessRunning": 0, "insertDate": "2017-09-30T15:13:23.359" }, "fields": { "insertDate": [ 1506784403359 ] }, "sort": [ 1506784403359 ]}
8
2.3.3. UmQueue
The UmQueue module gathers Universal Messaging channels and topics statistics. This helps know,on publish/subscribe processes, if queues are growing, or if the platform handles everything at thehighest possible speed.
Figure 12. UM queues length
Example JSON Data
{ "_index": "umqueue", "_type": "StartRouteMsg", "_id": "AV7TXczxh9aXLVWWGpYr", "_version": 1, "_score": null, "_source": { "name": "StartRouteMsg", "type": "channel", "currentNoOfConnections": 1, "currentNumberOfEvents": 0, "totalConsumed": 12, "totalPublished": 4, "usedSpace": 781, "memoryUsage": 0, "consumedRate": 0, "fanoutTime": 0, "insertDate": "2017-09-30T15:18:13.489" }, "fields": { "insertDate": [ 1506784693489 ] }, "sort": [ 1506784693489 ]}
9
2.3.4. UmDoc
The UmDoc module is the main module for business dashboards. It tracks every single (business)document published in the system.
Figure 13. CG Anomalies by type
Figure 14. Document published by type
10
Example JSON Data
{ "_index": "pubdoc", "_type": "DfArle", "_id": "AV7TTTGmh9aXLVWWGkmc", "_version": 1, "_score": null, "_source": { "DGE.Entete.Nature.Type": "DiffusionFichier", "DGE.Entete.Nature.Version": "02.01", "DGE.Entete.Nature.Phase": "DF_ARLE", "DGE.Entete.Service.Reference": "M0002E001", "DGE.Entete.Service.Version": "02.01", "DGE.Entete.Echange.Identifiant": "CNAM.DGEARLE.Q1785945", "DGE.Entete.Echange.Horodatage": "2017-09-30T17:00:04.911", "DGE.Entete.Echange.RefConversation": "DGEA011701", "DGE.Entete.Echange.Fonction": "9", "DGE.Entete.Emetteur.Identifiant": "99640000", "DGE.Entete.Emetteur.Type": "RFO", "DGE.Entete.Emetteur.Libelle": "DGE", "DGE.Entete.Recepteur.Identifiant": "01110021", "DGE.Entete.Recepteur.Type": "RFO", "DGE.Entete.Recepteur.Libelle": "CNAM", "DGE.Document.EchangeOrigine.Identifiant": "RGF011701", "DGE.Document.EchangeOrigine.Horodatage": "2017-09-30T17:00:00.203", "DGE.Document.Resultat.Statut": "R", "DGE.Document.Resultat.Complement[0].Code": "9302", "DGE.Document.Resultat.Complement[0].Libelle": "Adhesion producteur inconnue", "insertDate": "2017-09-30T15:00:05.157" }, "fields": { "DGE.Document.EchangeOrigine.Horodatage": [ 1506790800203 ], "ModuleCode": [ "null/null" ], "DGE.Entete.Echange.Horodatage": [ 1506790804911 ], "insertDate": [ 1506783605157 ] }, "sort": [ 1506783605157 ]}
11
2.3.5. Tablespace
The Tablespace module gives insights on Oracle TBS Occupation
Figure 15. Tablespaces occupation overtime
Figure 16. Tablespaces occupation gauges
Example JSON Data
{ "_index": "dbstat", "_type": "WEBMDATA", "_id": "AV7TUF8eh9aXLVWWGlge", "_version": 1, "_score": null, "_source": { "schema": "WEBMDATA", "allocatedMo": 200, "freeMo": 179.81, "usedMo": 20.19, "freePercent": 89.91, "usedPercent": 10.09, "insertDate": "2017-09-30T15:03:33.406" }, "fields": { "insertDate": [ 1506783813406 ], "usedPercentPretty": [ 0.10090000152587891 ] }, "sort": [ 1506783813406 ]}
12
2.3.6. Perflog
The perflog module listen to the cg-utils file that stores top level java services duration. With thisdata, you can see what services are taking too long to execute, and if the current duration isconsistent with the average.
Figure 17. Java service duration overtime
Figure 18. Java services duration heatmap
13
Example JSON Data
{ "_index": "logstash-2017.09.30", "_type": "perflog", "_id": "AV7TTUT9h9aXLVWWGknX", "_version": 1, "_score": null, "_source": { "offset": 76128, "method": "sendDfArle", "input_type": "log", "source": "/islogs/log4j2/perfs.log", "message": "2017-09-30 17:00:07.451 INFO PERFORMANCES - #sendDfArle(...): in 473.41ms", "type": "log", "tags": [ "beats_input_codec_plain_applied" ], "duration": 473.41, "@timestamp": "2017-09-30T15:00:07.451Z", "@version": "1", "beat": { "hostname": "c843aa83715b", "name": "c843aa83715b", "version": "5.5.2" }, "host": "c843aa83715b", "timestamp": "2017-09-30 17:00:07.451" }, "fields": { "@timestamp": [ 1506783607451 ] }, "sort": [ 1506783607451 ]}
14
2.3.7. Sysmem, Syscpu, Sysdisk (deprecated)
sysmem, syscpu and sysdisk provides insight of memory, CPU and disk usage from the serverhosting the current IS server.
They are deprecated since the unveil of the Elastic module metricbeat which is more feature-rich.
15
2.4. JenkinsThe Jenkins plugin Logstash gives insight on build launches and durations.
Figure 19. Jenkins builds duration
Example JSON Data
{ "_index": "jenkins", "_type": "build", "_id": "AV7PQahNh9aXLVWWB8FS", "_version": 1, "_score": null, "_source": { "data": { "id": "753", "result": "FAILURE", "projectName": "CNAV-DGE_P1_Review", "displayName": "#753", } }, "message": [ "[...truncated 18.72 KB...]", "[withMaven] Jenkins Task Scanner Plugin not found, don't display results of source code scanning for 'TODO' and'FIXME' in pipeline screen.", "[Pipeline] // withMaven", "[Pipeline] }", "[Pipeline] // stage", "[Pipeline] emailextrecipients", "[Pipeline] step", "[Pipeline] }", "[Pipeline] // timeout" ], "source": "jenkins", "source_host": "https://bpmfactory.s2-eu.capgemini.com/jenkins/", "@buildTimestamp": "2017-09-29T22:07:32+0200", "@timestamp": "2017-09-29T22:09:00+0200", "@version": 1 }}
16