FUTURESTACK13: Wake Up! A New Day Dawns for Alerts in New Relic from Bill Kayser, Distinguished...

Wake Up!A New Day Dawns for Alerts in New Relic

Bill Kayser

Wednesday, November 6, 13

You are always monitoring all the things.


WelcomeBill Kayser

@[email protected]

Covered in this talk:Overview of alerting in New RelicAlert Policies, other new featuresTips for getting the most out of alerts


mailto:[email protected]


What do we expect from alerting systems?


Photo credit: U.S. Coast Guard / Foter / CC BY-NC-ND


http://www.flickr.com/photos/coast_guard/8070997947/

http://www.flickr.com/photos/coast_guard/8070997947/

http://foter.com/

http://foter.com/

http://creativecommons.org/licenses/by-nc-nd/2.0/

http://creativecommons.org/licenses/by-nc-nd/2.0/

Moehre1992 CC BY-SA 3.0


Randy Chiu CC BY 2.0


Zinnmann CC BY-SA 3.0Wednesday, November 6, 13

http://commons.wikimedia.org/wiki/User:Zinnmann

http://commons.wikimedia.org/wiki/User:Zinnmann

What to Watch


What to Watch


What to Watch


What to Watch


What to Watch


Define Conditions


Define Conditions

Applications


Define Conditions

Applications

KeyTransactions


Define Conditions

Applications

KeyTransactions

Servers


What Action to Take


What Action to Take


What Action to Take


What Action to Take

37 Signals’ Campfire


What Action to Take



What Action to Take


Mobile Alerts


What Action to Take


Mobile Alerts


What Action to Take

37 Signals’ CampfireNotification Channels

Mobile Alerts



Thresholds and Notifications - Today














New in New Relic:Alert Policies


New Feature: Alert PoliciesAn Alert Policy specifies:

What to watchWhat action to take


Alert Policies


Alert PoliciesPolicies apply to:

ApplicationsKey TransactionsServers


Alert Policies


Alert Policies

Threshold settings and other configuration


Alert Policies


Notification channels


Alert Policies

Server policies are mapped to servers


Notification channels

ops-puppet-01


Application Alert Policy


Server Alert Policy


Key Transaction Alert Policy


Editing Alert Policies








Alert Policies

Named Policies


Alert Policies

Default Policy


Default Policies


Default Policies

Applications with defaults


Default Policies Default settings


Mapping Policies


Mapping Policies


Defining Channels


Defining Channels


Defining Channels


Defining Channels


Channel SettingsDefine distinct endpoints for each channel type


Channel SettingsDefine distinct endpoints for each channel type

Designate channels to be used strictly for downtime alerts


Notification GroupsDefine a set of notification channels for use across multiple policies

Include e-mail addresses, PagerDuty services, HipChat rooms, etc


Policies: Putting it All Together










Examining Alerts




What’s the Problem?Problems are events

Critical: tell me right away


What’s the Problem?Problems are events

Critical: tell me right awayCaution: note it in supporting detail



Alert




Alerts in the Event List








Alert History


Tips for Leveraging New Alerts


‘Alarm fatigue’worries hospitals

San Francisco Chronicle, October 23, 2013

“...a patient’s life could be hanging in the balance.”


Avoid Threshold SensitivityIncrease the lag time for triggering problems


Avoid Threshold SensitivityIncrease the lag time for triggering problems


Avoid Threshold Sensitivity

Increase the lag time for triggering downtime


Improve Threshold SensitivityAdd substring requirement to downtime settings


Improve Threshold Sensitivity

T = 800 ms

Apdex Score: 0.84 - 0.91

Mean Response Time


Narrow the ScopeDefine Key Transactions

Watch error rate and apdexCustomize apdex T value


Key Transaction Policies


Key Transaction Policies


Ignore Transaction NoiseJava: NewRelic.ignoreTransaction(), @NewRelicIgnoreTransaction

Ruby: newrelic_ignore [ :only => action..] [ :except => action...]

DotNet: NewRelic.Api.Agent.NewRelic.IgnoreTransaction();

PHP: newrelic_ignore_transaction ( )

Python: newrelic.agent.ignore_transaction(flag=True)


Stop waking me up


Stop waking me up(Wake up Jonathan instead)


Stop waking me up(Wake up Jonathan instead)


Alerts API


Alerts API


Alerts API


Alerts API


Alerts API


Alerts V2: Summary of ImprovementsAlert PoliciesNotification ChannelsNotification GroupsServer DowntimeLag time settingNew API


What do we expect from alerting? Get the right people involved at the appropriate time

Use well defined notification channelsGroup channels together for easy re-useTake advantage of PagerDuty

High signal, low noiseAlert on key transactionsEdit lag timesDistinguishing critical and caution problemsAdjusting apdex T


What do we expect from alerting Flexibility

Define policies according to specific operational requirementsUse defaults, define overridesAPI for integration, management and auditing

SimplicityWatch only the things you care about most


Migrating to use Alert PoliciesAccounts will be upgraded in November

One policy will be created for each application and key transactionOne policy will be created for each unique set of threshold settings across all servers

Existing alert behavior is preserved



Bill [email protected]@bravoking

Thanks!




Date post:	27-Jan-2015
Category:	Technology
Upload:	new-relic
View:	104 times
Download:	2 times

FUTURESTACK13: Wake Up! A New Day Dawns for Alerts in New Relic from Bill Kayser, Distinguished...

Technology