The IBM dashboard for operational metrics

transcript

Daniel KrookSenior Certified IT Specialist, IBM

We run Cloud Foundry on dozens of OpenStack VMs

Two intranet clusters

In the past year, we’ve learned how to

Classic: 38 huge VMs deployed with Chef: 1,302 users, 1,710 appsNG: 41 medium VMs deployed with BOSH: 123 users, 247 appsNot counting Dev deploymentsAll on 50+ Nova Compute nodes

• Keep Cloud Foundry running smoothly• Discover and prevent impending problems• Resolve unexpected issues quickly

1. Show the key data points we track

2. Show how our metrics dashboard helps us monitor that data

3. Share ideas on how to find better data in NG and beyond

4. Spark discussion on improved visibility for CF admins and customers.

Goals of this lightning talk

We are looking to get better at this, and help the community get better as well.

1. The key data

What are the important metrics?

Data that can be tracked over time to see trends and behaviors

Data that can help us predict problems before they happen

DEAs and apps health

Memory reserved as a proportion of the memory available

General health of all components

Health of the virtual machines Status of the processes running on them

Database nodes and services

Number of provisioned services against capacity available

At the PaaS layer, that means:

Deliver continuous availability in the cloud

Proactively solve problems rather than react to them

Understand the behavior of the system to automate it

Why do we need metrics?

NATS message bus• Discover the components to interrogate • Best for dynamically changing data

Where can we find them?

Cloud Controller database (CCDB)• Longer lived data that isn’t in the varz endpoints

2. Monitoring that data

1. Views of component health

2. Resource usage details

3. Ongoing growth trends

4. Access to logs and raw varz

5. Email notifications

Our metrics dashboard provides…

Components nearing capacity or failure Already failed components Out of control apps and noisy users

Active/inactive users and apps Growth trends and runtime/service adoption

It helps us find (and fix) problems

It helps us see patterns

User and app trends

There is also one unauthenticated page for high level stats

DEA list

DEA details

Service node list

Service node details

User list

User details

App list

App details

Log list

Log details

Email notifications

3. Finding and acting on better data

NG provides granular user/org/space views…• This enables better BSS potential in terms of QoS and departmental billing

…But we lost user and app data linkages from the health manager• Can’t see what DEA my app resides on (not currently enabled in our NG version)• Can’t see how many apps a user has (replaced by orgs and spaces, but still

valuable to trace)• See https://github.com/cloudfoundry/cloud_controller_ng/issues/81

We’d like to restore that data, either surface it • in varz endpoints (dynamic data, preferred) or • CC_DB (static data, could be a security concern)

Let’s resolve gaps in data captured from NG

Detect errors in applications that are traceable to users/orgs• Preemptively reach out to them to see if they need help• Think customer service and proactive support!• Can we hook into to BOSH or Jenkins for automation?

Automate (and expand links to the IaaS and SaaS stacks)• Self healing systems (out of disk, move apps)• Self scaling systems (detect when nearing thresholds)• Evolving topologies (replace unused service nodes with popular ones)

Let’s begin to link metrics to automation

Admins are the primary beneficiary right now• But data is almost completely read only• Should we provide UAA based tiers of access to admins?

Others can and should benefit• Customers

• End users• Developers

• Management• Executives, line of business owners• Finance

Let’s expand the broadcast of metrics to more users

Thanks!

The metrics dashboard innovators

Chris Peters Russell Boykin

Doug Davis Wei Feng

We’re hiring!

Search Jobs at IBM by:

SmartCloud Application Services

The IBM dashboard for operational metrics

Technology