1

Proactive Monitoring for PowerCenter:

What’s Next in Governance and Operational Environmental Monitoring

Chris Carlson

Director, Product Evangelist

Informatica

#IW2012

2

Agenda

• Overview of Proactive Monitoring for

PowerCenter

• Background of the Solution

• Technical Overview

• What’s Next in Operations and Governance

• Product Demonstration

3

Overview

• Capabilities and features of Proactive Monitoring for

PowerCenter Operations and Governance

• What’s new: environmental variables (CPU, memory, etc.) and

governance rules (enforcing best practices)

• How to use Proactive Monitoring to achieve faster response

times to potential operational and governance issues in your

PowerCenter environment

The Proactive Monitoring for PowerCenter solutions have dozens of prebuilt

rules and pattern logic combined with the flexibility of self-service that allows

users to easily modify applications without going through long cycles.

These solutions address a diverse array of potential issues, including session

and workflow breakdowns, trending data issues, and the vast area of

development governance.

4

Why did we Build this? What Are Some Customer Pains?

“We invest a lot on resources (manpower) to monitor

PowerCenter – and we still miss many issues”

“We have tools to report on performance after an

issue occurs, but nothing that helps us prevent it”

“I want out of the custom scripting business for

monitoring. Not scalable and not effective anymore”

“We have no way at all to enforce best practices”

5

Informatica Built these Solutions to Address Three Common Customer Concerns…

1. Customers asked for a “complete, PowerCenter-centric” Monitoring Solution

2. Customers want to become “Proactive” in seeing issues before they turn into problems. “Smart push” the alerts instead of static dashboard displays.

3. Scalable way to identify anomalies such as Workflows running longer/shorter than normal or returning more/less rows than normal.

6

Time Spent on PowerCenter Deployments

Development

Testing

Monitoring of

PowerCenter Workflows

and

Environment

Best

Practices

Where is your time spent? ROI Associated with PMPC

• 70% Reduction on Time Spent Manually Monitoring Workflows

• 80% Reduction on Time Spent Manually Monitoring Best Practices

• Soft ROI • Reduced Help Desk Costs • Avoid Issues • Increased Reliability

Regain substantial time for new development, new projects, etc…

7

What is Proactive PowerCenter Monitoring?

Prebuilt Solution to Monitor PowerCenter

Delivers Alerts to people, dashboards and systems

Complete Operational and Best Practices

Extensible – build

unlimited number of new rules

8

Move from Reactive to Proactive

Time

Event(s)

• Workflow is started

• Changes are made

• Workflows are Named

• Paths are hardcoded

Issue(s)

• Workflows run long

• Developers make many

changes

• Source rows rejected

• Development Practices

slip to Production

Problems and Losses

• Schedules are missed

• No rows are loaded

• Zombie Workflows

• Workflows Fail

• Phone Calls At 3am

• Business doesn’t get

reports

Proactive Reactive

9

Customer Case Study

PowerCenter

Continuous growth in complexity

Case Study: A media enterprise

No of folders: 112 No of schedules: 6,336

No of mappings: 18,180 Work flow executions per day:

3,310

No of transforms: 93,940 Session executions per day:

10,402

No of sessions: 21,879

No of workflows: 13,109

One Shared Service Environment

Challenges:

• Existing monitoring tools did

not look at PowerCenter

internals

• No Possible Way to Monitor

over 10,000 Sessions Per Day

with existing tools or manually

Solution:

• Proactive PowerCenter for

Operations monitors 24X7

in a “Proactive Manner”

• Can now catch data issues

like “less data” than

expected because of

completeness of solution

10

Solution Overview

Proactive PowerCenter Monitoring

License-Limited + Rule Pack

Alert Enrichment

DBs, Web Services and more…

Alert Editing, Creation and Management

Custom: Environmental (CPU, Memory..)

DB (SQL)

JMS Web Services

• Operational

• Design Time

• Start/Stop Workflows

• Create Help Desk Ticket

• Update Control Tables

SQL Sources to Expose Repository Features and Allow Easy Rule and Report Building Environmental Monitoring Runtime Statistics Receiver Ping PowerCenter Services

Notification Framework for Alerts Prebuilt EMAIL Responses Prebuilt Dashboard Alerts PowerCenter Workflow Response to Execute Commands

11

Component Architecture

Monitor & Alert

Proactive Monitoring

for PowerCenter (Source Feeds, Rules/Templates,

Watchlists, Alerts)

Workflow

Controls

Alerts

(Dashboards, e-mail, DBs, other systems, etc)

Real time Workflow and

Session Data PowerCenter

Runtime

Workflow

Statistics PowerCenter

Repository

PowerCenter Server (CPU,

Memory)

PowerCenter Repository

(Table space, DB Ping)

Environment

Monitoring

12

Alert Scenario Categories

Category Scenarios

Stop Light - Are Domain services running?

Breakdowns - Did a Session/Workflow fail? When?

- Did something recently change?

- Are there design-time artifacts in

production (e.g. hardcoded paths)

Data Issues - Failed rows in Session/Workflow

- Succeeded Workflow with Zero

records loaded

Trending - Workflows deviating from norm

(defined or dynamic threshold)

Environmental - Failures correlated with CPU load,

memory usage, process

13

Proactive Monitoring for PowerCenter Operations Alerting Rules & Templates (ships with over 23 scenarios)

Operational Monitoring

1 Workflow/session was successful but zero

(0) records loaded

2 Rejected Records in Session

3 Process running on integration service node

with more than X% CPU usage

4 Repository service database table space

usage is more than X%

5 Session/Workflow Failures

6 SLA Violations: Workflow/Folder/Repository

Level

7 Significant increase in Workflow/Session

Elapsed Times

8 Significant decrease in applied row counts

9 Workflows missing schedules

10 Too many concurrent workflows

11 Automatic Restart of Zombie Workflows

12 Ping Domain/Integration Service/Repository

Service

Plus optional Environmental Variables:

Node Operating System CPU, Memory , and Processes

PowerCenter Repository Database – Heartbeat and

Tablespace checks

• Identify top memory and CPU consumers

• Correlate failures to environmental issues

• Faster identification of root causes

• Build new alerts for any number of correlations

14

Sample Alert Missed Schedule Enriched with Environmental Data

15

Ensuring the Right People get the Right Amount of Alerts

• Targeted Alerting

• Administrators

• Operations

• Developers

• Architects

• Set Alerts on…

• Folders

• Repository

• Domain

• Workflow

• Session

• Transforms

16

Integrate Alerts with your IT processes

• Multiple output channels (one, many, all):

• Send email to user / distribution group

• SMS TXT, pager

• Instant Message alerts (XMPP)

• Trouble ticketing systems

• Network management systems

• Dashboards and reports

• Alert management features:

• Alert aggregation, suppression, snooze

• “Do not disturb” / exception lists

• URL Link to knowledge bases and troubleshooting steps for 1st/2nd level support personnel

• Configure to support monitoring for alert acknowledgement

17

Proactive PowerCenter Governance: What is it?

It is a framework to implement alerting and reporting to monitor

best practices related to the use of

(Transform/Session/Workflow) attributes to control the

behavior of PowerCenter

• Who can use it?

• Managers of PowerCenter Development teams

• PowerCenter Administrators

• Which attributes are covered?

• Almost all the attributes that are configured through PowerCenter Designer and Workflow Manager (in the Properties and Session Config tabs) are covered

• How many attributes are there?

• Approximately 250 attributes

18

Proactive Monitoring for PowerCenter Governance Alerting Rules & Templates (future)

Governance & Best

Practices Naming Conventions:

Mappings/Sessions/Workflows

Invalidated Mappings/Sessions/Workflows

Transforms saved with Default Values

Saving objects without comments

Frequent Changes to Mappings/Sessions

Hardcoded Paths in Object definitions

Illegal Command tasks used in Workflows

Duplicate Mappings/Sessions/Workflows

Customizable Session Configuration alerts

License allows for limitless rules

Initially we created a set of “High Level” Best

Practices Rules

Templates with over 250 pre-mapped attributes makes

creating governance rules fast and effective.

19

Top Governance Attribute Alerts (that can impact production)

• Lookup SQL Override is set

• Reset option in Sequence Generator is checked

• Tracing is set to Verbose

• Commit Interval is set too high

20

Scheduled Governance Reporting

21

Demonstration

• Run Time Failure after Design Time Change

• Workflows Runs Longer than its past Average

and Misses SLA

• Design Time Best Practices Enforcement***

• Build a New Throughput Rule Using the

RulePoint Wizard

22

Informatica Marketplace Overview A Trusted, Open Ecosystem

• Virtual Marketplace for Data Integration Apps

• Solutions across all technology areas – DI, DQ, MDM, Cloud, etc.

• Open Ecosystem – Apps from Partners, ISVs, Consultants, and Developers

• Seal of Approval ensures App quality

• More than 600 Apps, over 200 Free!

• 15k visits per month, 2k downloads

http://marketplace.informatica.com

R

M

23

Thank You