Date post: | 17-Dec-2015 |
Category: |
Documents |
Upload: | edith-black |
View: | 218 times |
Download: | 3 times |
Application Performance Monitoring
One Approach
John Slobodnik
April 18, 20061:30 p.m.
CMG Canada
Introduction of Product Introduction of Product SuiteSuite
• ServerVantageServerVantage
• ApplicationVantageApplicationVantage
• ClientVantageClientVantage
• VantageAnalyzerVantageAnalyzer
• VantageViewVantageView
ServerVantageServerVantage (SV)(SV)• Collects “server” level data.Collects “server” level data.
– Multiplatform: Windows, LINUX, UNIX, etc.Multiplatform: Windows, LINUX, UNIX, etc.– CPU, memory, disk, network out of the box.CPU, memory, disk, network out of the box.
• Collects “application” level data.Collects “application” level data.– Oracle, SQL server, WebLogic, IPlanet, LDAP, etc.Oracle, SQL server, WebLogic, IPlanet, LDAP, etc.
• One SV agent installed on each client server.One SV agent installed on each client server.– Runs most of the time.Runs most of the time.
• Customized counters (metrics) can be written.Customized counters (metrics) can be written.
ApplicationVantage (AV)ApplicationVantage (AV)
• A sniffer.A sniffer.– Agent-based application analysis of packet level Agent-based application analysis of packet level
communications.communications.
• Gathers all traffic that passes through the Gathers all traffic that passes through the Network Interface Cards (NICs).Network Interface Cards (NICs).
• Can merge the data together from multiple Can merge the data together from multiple servers.servers.– Can trace, for example, SQL server traffic.Can trace, for example, SQL server traffic.
• One AV agent is installed per client server.One AV agent is installed per client server.– Turned on when required.Turned on when required.– Most often in firefighting mode.Most often in firefighting mode.
ClientVantage (CV)ClientVantage (CV)
• Gathers data on the performance of your Gathers data on the performance of your application.application.– Done through timings of synthetic business Done through timings of synthetic business
transactions on CV workstations (robots).transactions on CV workstations (robots).
• Scripting of business transactions done Scripting of business transactions done with a tool called QARun. with a tool called QARun. – We are doing active monitoring.We are doing active monitoring.– There are two other options available here There are two other options available here
now:now:• Passive monitoring using CVPassive monitoring using CV• A hardware-based solutionA hardware-based solution
Vantage Analyzer for J2EE Vantage Analyzer for J2EE (VA)(VA)• J2EE (Java) based tool to help pinpoint exact J2EE (Java) based tool to help pinpoint exact
locations of code-level performance problems.locations of code-level performance problems.– Locates slow methods, SQL statements and transactions.Locates slow methods, SQL statements and transactions.
• The VA agent runs inside your Application servers The VA agent runs inside your Application servers JVM obtaining performance metrics using Byte JVM obtaining performance metrics using Byte Code Instrumentation.Code Instrumentation.– Data sent in real-time to nucleus server where it is Data sent in real-time to nucleus server where it is
stored and distributed to VA performance consoles.stored and distributed to VA performance consoles.
• Supports WebLogic, Websphere, etc.Supports WebLogic, Websphere, etc.• Customized componentCustomized component
– Allows a transaction to be followed in VantageView.Allows a transaction to be followed in VantageView.
VantageView (VV)VantageView (VV)• Web-based portal for viewing SV, CV, AV Web-based portal for viewing SV, CV, AV
and VA data for monitoring and reporting. and VA data for monitoring and reporting. • Accessing information from the Vantage Accessing information from the Vantage
suite of tools, VantageView users check suite of tools, VantageView users check the status of clients, servers and networks the status of clients, servers and networks from their intranet while also providing a from their intranet while also providing a near-time service level perspective on near-time service level perspective on application availability and performance. application availability and performance.
• The flexibility of VantageView enables The flexibility of VantageView enables different levels of users to view pertinent different levels of users to view pertinent information for easy problem information for easy problem determination and resolution.determination and resolution.
• Customized counters (metrics) can be Customized counters (metrics) can be created in the VV database.created in the VV database.
`
`
`
`
`
``
`
`
`
`
`
VantageView
ServerVantage ClientVantage 1
ClientVantage 2
ApplicationVantageVantage Analyzer
NucleusVantage Analyzer
SQL DB
Application #1SV Agents
Application #2SV Agents
Application #1CV Agents
Application #2CV Agents
Application #3CV Agents
Application #1AV Agents
Application #2AV Agents Application #1
VA Agents
Control Servers
Database Database
Database Database
Database
Monitored Servers
A Few Easy Setup StepsA Few Easy Setup Steps• A summary of the steps to implement the
solution:– Install the agents.– Complete Administration– Set Preferences– Management
• Create tasks and apply blackout schedules.– Create monitoring views.– Create reports.
• Optional steps taken– Create dashboards.– Create custom counters (metrics).
Install the AgentsInstall the Agents
This is a quick, procedural task that This is a quick, procedural task that is quick to complete.is quick to complete.– A script is run to do the install followed A script is run to do the install followed
by any applicable patches.by any applicable patches. The product keeps track of the level The product keeps track of the level
of agent installed on each server in a of agent installed on each server in a central repository.central repository.
SV and AV agents are installed on SV and AV agents are installed on each server (Window, UNIX, LINUX)each server (Window, UNIX, LINUX)
Administration – Administration – Configure DatabasesConfigure Databases
Set up the online database(s).Set up the online database(s).
Configure Historical DatabaseConfigure Historical Database
Define the historical database.Define the historical database.We keep 3 months of data online.We keep 3 months of data online.All else goes to historical All else goes to historical database.database.
Control Server ConfigurationControl Server Configuration
Set up the control servers.Set up the control servers.
Define UsersDefine Users
Define VV user profiles.Define VV user profiles.
PreferencesPreferences
Business Business applicationsapplications
4 applications.4 applications.
Business locationsBusiness locationsVarious Various Canadian Canadian cities.cities.
Business transactionsBusiness transactionsAn application (29 An application (29 transactions) is transactions) is broken down into 3 broken down into 3 transaction transaction groupings (14, 7, 8).groupings (14, 7, 8).
Server groupingsServer groupingsProduction, pre-Production, pre-production, support, production, support, third-party, etc.third-party, etc.
Management - Create TasksManagement - Create Tasks
Create a new task.Create a new task.
Create TasksCreate Tasks
Select the type of server: Windows, UNIX, etc.Select the type of server: Windows, UNIX, etc.
Create TasksCreate Tasks
Select the counters you wish to Select the counters you wish to see.see.
Create TasksCreate Tasks
Add a rule for alerting.Add a rule for alerting.
Create TasksCreate Tasks
Set up alerts if you want them. For example: System Thrashing, Set up alerts if you want them. For example: System Thrashing, TCP Connectivity lost from WL to WL layer, CPU > 90%, etc.TCP Connectivity lost from WL to WL layer, CPU > 90%, etc.
Create TasksCreate Tasks
If you can do it from a command line it can be automated here: shell scripts, If you can do it from a command line it can be automated here: shell scripts, bat files.bat files.Perform an action based on a threshold being breached.Perform an action based on a threshold being breached.(1) Kick off a WL thread dump based on a WL counter below a certain level.(1) Kick off a WL thread dump based on a WL counter below a certain level.(2) Send an alert based on an ASCII pattern match.(2) Send an alert based on an ASCII pattern match.(3) Previous problems can be proactively addressed with this type of (3) Previous problems can be proactively addressed with this type of instrumentation. We examine WebLogic logsinstrumentation. We examine WebLogic logs
Alert notification via pager, email, SNMP, etc.Alert notification via pager, email, SNMP, etc.Different audiences for different tasks, DBAs, App. Different audiences for different tasks, DBAs, App. Support, etc.Support, etc.
Create TasksCreate Tasks
Select the appropriate data sampling interval.Select the appropriate data sampling interval.Key to the size of your database.Key to the size of your database.
Create TasksCreate Tasks
Start the task.Start the task.
Management – Blackout Management – Blackout ScheduleSchedule• Apply a blackout schedule, if applicable.Apply a blackout schedule, if applicable.
– ServerVantage agents do not run when the ServerVantage agents do not run when the application is down daily.application is down daily.
• Client Vantage robots are also set up to Client Vantage robots are also set up to run on a blackout schedule.run on a blackout schedule.– Implemented through CV which uses the Implemented through CV which uses the
Windows scheduler.Windows scheduler.
Create Monitoring ViewsCreate Monitoring Views
Monitoring views contain all data points.Monitoring views contain all data points. Flexible: you can plot many different Flexible: you can plot many different
metrics on the same chart.metrics on the same chart. Business metric vs. server performance.Business metric vs. server performance. Application metrics vs. server metrics.Application metrics vs. server metrics.
TeeChart Editor gives you Excel chart TeeChart Editor gives you Excel chart type functionality to modify the look of the type functionality to modify the look of the chart. chart.
Monitoring ViewMonitoring View
Saved as a permanent monitoring view report.Saved as a permanent monitoring view report.
Monitoring ViewMonitoring View
Monitoring (ad hoc)Monitoring (ad hoc)
Can drill into data point.Can drill into data point.
Drill into IDPDrill into IDP
Intelligent Data Point (IDP)Intelligent Data Point (IDP)
Create Reports
• Reports contain different levels of data summarization.• From all data points to daily average.• We have created 12 hour, 2 day, 1
week and 1 month views of all reports.• Flexible: you can plot many different
metrics on the same chart.• TeeChart Editor gives you Excel chart
type functionality to modify the look of the chart.
Create Reports
Select the metric source.Select the metric source.
Create Reports
Select the metric(s) desired.Select the metric(s) desired.
Create Reports
Select the time range.Select the time range.
Create Reports
Select the display format.Select the display format.
Create Reports
Schedule the report.Schedule the report.
Create Reports
Save the report.Save the report.
Reports
Reports
Reports
Then the Business asked…Then the Business asked…
How can we prove that the API calls are How can we prove that the API calls are performing better?performing better?
Custom program installed on WL servers.Custom program installed on WL servers. Gathers API call response time data, converts it to a Gathers API call response time data, converts it to a
local CSV file, FTP to VV database.local CSV file, FTP to VV database. API Response Time report created, queries VV DB.API Response Time report created, queries VV DB. APIs split between internal vs. outsourced (for reporting APIs split between internal vs. outsourced (for reporting
purposes).purposes). There are a number of activities within each bean There are a number of activities within each bean
conversation.conversation.
API Response Time ReportAPI Response Time Report
Sample bean conversation report.Sample bean conversation report.
Then Management said…Then Management said…
We need to have a some different We need to have a some different dashboard views.dashboard views.Each level of dashboard gets more detailed.Each level of dashboard gets more detailed.Special dashboard for outsourced Special dashboard for outsourced
infrastructure.infrastructure.Dashboards were created using the Dashboards were created using the
integrated VISIO (Vantage Visualizer) integrated VISIO (Vantage Visualizer) piece of the product.piece of the product.
Management DashboardManagement Dashboard
Drill down to Application AvailabilityDrill down to Application Availability
Application Availability (bottom)Application Availability (bottom)
Drill down to Heat ChartDrill down to Heat Chart
Drill down to CNS reportDrill down to CNS report
Drill down to Application ScorecardDrill down to Application Scorecard
Application Scorecard (bottom)Application Scorecard (bottom)
Drill down to Transaction ScorecardDrill down to Transaction Scorecard
Drill down to Performance SummaryDrill down to Performance Summary
Drill down to Orders ReportDrill down to Orders Report
Drill down to Session Current Drill down to Session Current Count reportCount report
Drill down to WL Serviced Drill down to WL Serviced Requests reportRequests report
Geographic DashboardGeographic Dashboard
We asked ourselves…We asked ourselves… How can we make this easier to support?How can we make this easier to support?
Customized metrics can be created in VV or SV.Customized metrics can be created in VV or SV. Make non-standard types of metrics available.Make non-standard types of metrics available.
Samples of some of the customization created:Samples of some of the customization created: Disk usage of SV logs files directory.Disk usage of SV logs files directory. Automated removal of SV log files. Automated removal of SV log files. Automate push of patches to all agents.Automate push of patches to all agents. Send a command to run on a server and return the result.Send a command to run on a server and return the result. Count the number of SV datafiles.Count the number of SV datafiles. Agent restart.Agent restart. Gather SV log files.Gather SV log files.
More CustomizationMore Customization TCP Connection test from WL layer to WL layer.TCP Connection test from WL layer to WL layer. Number of Orders.Number of Orders.
SQL query to xml to csv to VV DB.SQL query to xml to csv to VV DB. ASCII file pattern match in WL logs (3).ASCII file pattern match in WL logs (3).
Automatic thread dumps WebLogic.Automatic thread dumps WebLogic. Average Elapsed TimeAverage Elapsed Time
Customer purchase at store experience.Customer purchase at store experience. Individual transaction timings is CV, adding them up is custom.Individual transaction timings is CV, adding them up is custom.
Network Test / TCP Connection TestNetwork Test / TCP Connection Test Traceroute response time for up to 10 hops & alert.Traceroute response time for up to 10 hops & alert.
API Response time monitoring.API Response time monitoring. Average, max, min, std devAverage, max, min, std dev
Network Connection TestNetwork Connection Test
Vantage AnalyzerVantage Analyzer
• Installed on production WebLogic servers during the peak annual sales period.
• Now in the pre-prod environment.– So bugs can be found before promoting new
code to production.
J2EE JavaScapePaints a landscape view of your J2EE environment. This view displays component interactions between JSPs, Servlets and Web services, Session, Entity and Message-driven Beans, as well as database usage.
Transaction ExplorerTransaction ExplorerThe tree is organized by the largest consumers, from top to The tree is organized by the largest consumers, from top to bottom. The tree can be sorted by the CPU or Transaction time bottom. The tree can be sorted by the CPU or Transaction time period.period.
Transaction ScopeTransaction ScopeGives a detailed view on each individual transaction which runs Gives a detailed view on each individual transaction which runs through your application.through your application.
Stalled ThreadsShows thread-level detail of a transaction.
Method HotSpotsMethod HotSpotsIdentifies the biggest consumers in your application. The view Identifies the biggest consumers in your application. The view can be sorted by Transaction or CPU time.can be sorted by Transaction or CPU time.
SQLyzer HotSpotsSQLyzer HotSpotsLets you pinpoint the largest SQL consumers.Lets you pinpoint the largest SQL consumers.
SLA MonitoringSLA MonitoringThis view displays pre-configured SLA rules and when they were This view displays pre-configured SLA rules and when they were last violated.last violated.
Memory HotSpotsMemory HotSpotsLocate memory leaks as well as memory allocation hot spots to Locate memory leaks as well as memory allocation hot spots to help assist with server availability and performance.help assist with server availability and performance.
SummaryManagement extremely pleased.
Customized dashboards, peak period success, want more application’s instrumented.
Business application ran almost 99.9% availability during peak processing period of the year in large part due to this solution.
Now instrumented to be more proactive than in the past.Being used as a model for the rest of the enterprise.
Support teams have embraced the solution because it makes their lives easier.
DBA’s, application support, system administrator’s, performance and capacity planners, etc.
Significantly less time wasted determining whose problem it is (you know, 6 teams in a room…) during fire-fighting.