QueueMetrics - Tips and Tricks

transcript

Presented by:Lenz EmilitriFounder, Loway@lenz

EffectiveCall-Center monitoring and reporting

QueueMetrics Tips and Tricks

Asterisk Africa – Johannesburg, Feb 16, 2016

Contents

●Introduction

–Who we are

–What we do

●Running QueueMetrics

●Integrating QueueMetrics

●Open discussion

Who we are

Since 2004, Loway is leading the way in development of advanced software solutions for

the Asterisk® PBX.

With QueueMetrics we set up modern standards for call center performance

measurement.

Our Mission is to put Swiss passion for precision and reliability at customers' service.

Together with Loway, you can provide your clients with the most reliable, flexible, and

sophisticated call-center management solutions available today.

What we do

Our History:

● Started working with Asterisk in 2003● Developed QueueMetrics in 2005● Developed WombatDialer in 2012● Launched QueueMetrics Live in 2015

Installed base:

QueueMetrics currently deployed in thousands ofcall-centers worldwideAverage site: ~50 agentsLargest sites: ~1000 agents live (on Asterisk clusters)

WombatDialer deployed in ~300 sitesAverage site: ~80 channelsLargest site: ~4000 channels

Client base:

● 30% USA / Canada● 25% Europe● 20% LATAM● 10% Africa● 10% Asia● 5% Middle East

Contents

●Introduction

–Monitoring & Performance

–Configuration best practices

–Extending call tracking

–Quality Assessment (QA)

●Open discussion

Contents

●Introduction

●Open discussion

The key process to improve run-time performace is:

Monitoring → Assessing → Understanding → Fixing.

●Monitoring builds a baseline of data

●Assessing lets you define how the problem appears

●Understanding – that's the hard part!

●Fixing is often trivial.

●Simply throwing better hardware at the problem will not usually help in

the long run.

●Don't wait for problems to start monitoring!

Monitoring & Performance

QueueMetrics was designed to be run in call-centers with up to

1,000 live agents.

Most performance problems in mid-sized CC's (up to 150 agents)

are caused by:

●Java memory (mis)configuration

●QM data caching turned off

●Broken MySQL indexes

●Wrong audio storage model

●JSON or XML-RPC wallboards

Performance problems?

Monitoring – Java Memory

QueueMetrics is written in Java.

●Java uses a fixed memory

pool that you need to configure.

●Target: no stop-the-world major collections. That's where the

system appears to hang.

●The RPM defaults are NOT okay for larger systems

●Sensible defaults:

-Xms4096M -Xmx4096M -server -XX:+UseParallelOldGC -

XX:PermSize=512M -XX:MaxPermSize=512M

●But you need to monitor them!

It is very easy to monitor remotely a production JVM.

Monitoring - jVisualVM

●Check heap size and parameters

●Monitor GC activity vs CPU usage

●Monitor threads:

–1 thread ~ 1 active request

–>5 users per thread

●Normal for memory to go up until GC'd

●Thread dump: what is QM doing now?

–Standard QM scripts let you take thread dumps

●CPU Sampler: why is it taking so long?

Internal tools: Admin page → System diagnostic tools

Monitoring – QM internals

RAM caches

–Data for all partitions

–SQL cache efficiency > 90%

–Cached strings < 1M

Live inspector

–See data being loaded

–Recent activity

–Quite expensive to run

Sometimes we find random MySQL issues.

●Symptoms

–Performance degrades strongly and all of a sudden

–Very high MySQL CPU usage and light Java CPU usage

–Lots of open threads

–Very high disk I/O on MySQL server

●Root cause

–Access indexes on tables present but broken / unused.

●What to do:

–Manual reindexing (drop indexes and recreate)

–Might take a while on very large databases

–..assuming I/O on the server is adequate (SSD!)

Monitoring – MySQL issues

If your audio recordings are all stored in a single folder, scan times can

get high...

●Symptoms

–QueueMetrics very slow when opening the call pop-up window

●Root cause:

–Improper file storage

●Solution

–Split your recording in multiple folders, per day and queue, e.g./mount/audio/2016-01/30/queue1/….

–Use LocalFilesPerDay as the audio playback PM

You can also have multiple locations scanned in sequence, e.g. local

first, NAS next.

Monitoring – recordings

If you use a remote wallboard or any other piece of integration though JSON

or XML-RPC webservices, it should be well behaved.

●Symptoms

–Overall performance degradation

–VisualVM shows tons of open threads.

●Root cause

–Too many requests sent per unit of time

●Solution

–Services should cache local results if they need to feed multiple consumers / wallboards at once

–Services must wait for an answer before submitting another – if not, it's sure „Death by a Thousand Threads“.

Monitoring – Services

QueueMetrics offers a plain service to be used by

monitoring apps:http://my.qm:8080/queuemetrics/sysup.jsp

●Lightweight JSON response

●Checks memory and DB connectivity

●Easy to integrate with existing monitoring platforms (Influx,

Riemann, Zabbix… you name it)

Monitoring – Sysup

We run hundreds of QueueMetrics instances in our service

QueueMetrics Live.

Monitoring – what we do

●Chose very basic metrics:

–total memory,

–free memory,

–web hits

–loader hits

–errors

●Easily spot 95% of issues

●Manually instrument if

needed

Contents

●Introduction

●Open discussion

QueueMetrics is built around an all-encompassing security model

based on keys:

●System keys (access QM functions and pages)

●Custom keys

Users have a keyring:

–Default keyring = security class

–Can add an revoke keys for each user

–Masterkey

●All functionalities have a required system key

●All objects have optional keys ( = locks)

–Blank key = visible to everybody

The security model

Create different classes for different kinds of users

●Different clients

●Different access roles

●Start with the default ones (AGENT, USER, etc.)

Edit keys with the new editor:

●Plain-English view

●Set individual keys

●View inherited keys

Do not use the master key

Security best practices

ConfigurationAll configuration stored in the system folder.

●You can see which one is used in the License page

A text file holds the current configuration keys:

●File: configuration.properties

●General defaults

●General settings

●Picked up on user log-in

A different configuration file holds licensing information:

●File: tpf.properties

●Picked up on system restart

Config. best practicesKeep a backup of your current configuration!

Edit the configuration.properties file through Home → Edit system

parameters.

Edit common properties through the Explorer GUI: Home → Explore

system parameters

View the current configuration through the Home → System

Diagnostic Tools → View Configuration.

QueueMetrics was born to analyze inbound queues...

●In order to analyze outbound, we used special „outbound queues“

that used a piece of dialplan to do the tracking

●When you do outbound in a call-center context, you need a

„reason“ to distinguish calls

● You can originate outbound calls through phones or though the

Outbound calls in QM

For best results...

●Use an underlying physical queue for outbound.

–Presence

–Pauses

–Hotdesking

–No calls!

●Name the queue „q-XXX“ where „XXX“ is the inbound queue

●Use the Icon page to originate, not phones

●If you need to place a lot of calls, use a dialer solution that works

with QM (e.g. WombatDialer)

Outbound – best practices

Agent configuration

Agents are used by QueueMetrics to

map names and properties to a PBX

code. An agent has...

●An agent code

●A name

●A supervisor

●A group

●A set of queues

●Etc...

Users are used by QueueMetrics to allow

access to the GUI.

A user has...

●A login

●A password

●Belongs to a class

●Has a set of security keys

●Etc...

What is the difference between an Agent and a User?

In order to let an agent use the GUI, its Login must match the Agent code (e.g. Agent/101)The QueueMetrics licensing model is based on Agents, not Users.

Agents – best practices

For best results...

●Use simple SIP phones

–Configure max lines = 1

–Check agent state on queues

●If using Local channels, avoid excess dialplan

●Use hot-desking

●Use the Icon page

–Easy log-on multiple Q's

–Easy status tracking

–Agent messages

–Outcomes

Contents

●Introduction

●Open discussion

Asterisk logs the basic events for queue. QueueMetrics improves on this,

by tracking:

●Start of call (as opposed to queueing of call)

●DNIS

●IVR traversal and menus

●IVR goals

●Music on Hold

As they are not provided by Asterisk, they need to be added at the

dialplan level by adding INFO records.

Tracking more information

By producing a log entry like the following:

1353461650|1353461627.33271|NONE|NONE|INFO|IVRSTART|1234|5556777

You get:

●Actual start of call

●DNIS

●Call visible on RT before it's queued

Tracking DNIS

QueueMetrics is able to track the full life cycle of a call.

You need to:

●Send events when keys are pressed

●Send events when goals are reached

●Queueing is automatically considered a goal

Tracking IVRs

How often were your IVR

menus traversed?

When and where did people

How many self-service goals

were reached?

Tracking IVRs: Reports

How much did it take to

traverse each menu?

For many call-centers, measuring Music-on-Hold on answered calls is

important:

●Directly affects the perceived quality of service

●Anomalous MOH durations are an „alarm bell“ for underlying

problems

●QueueMetrics fully supports it

–Specific reports

–Call details

So… where is the catch?

●Asterisk does not produce these events for queues

Tracking Music-on-Hold

So how do we do it?

●Patches available for 1.4 & 1.8

–Never accepted into Asterisk

–Used in production in a number of 100+ channel CCs for years

–Hard to upgrade

●Passive monitoring daemon - MohTracker

–Uses AMI and works with any version of Asterisk

–Available as alpha software, free of charge

–Contact us if interested

Tracking MOH (2)

Contents

●Introduction

●Basics of QA monitoring

●Effective scoring

●Performance tracking

Open discussion

Contents

●Introduction

Open discussion

Quality Matters!

●Track calls

–Find issues before they find you.

●Track agents

–Improve training

–Assess strenghts and weaknesses

●Track queues

–Are we doing what we are expected to be doing?

–Can we show it?

If you are not using QA now, you are losing out.

QA – Introduction

Each call is graded on a set of metrics you define

–Each metrics has its own Engagement Code

–They map a numeric score (0-100)

–Same metric be used in multiple forms

Metrics are grouped into Forms

–Up to 10 sections with up to 130 questions

–Metrics are grouped into four grade bands (Issue, Req.Impr, Meets exp., Exceeds exp.)

–For each form, a score is computed

–Forms are immutable (more or less)

–Each call can be graded on multiple forms

QA – General ideas

●Form input is flexible

–Questions can be hidden or shown

–Questions can be weighted in order to form a score

–Shortcut questions

●Immediate agent feedback

–Receive a task when a call is scored

–Agents are immediately engaged

–Agents must acknowledge or dispute it

QA – General ideas (2)

QA is a process and requires some planning.

●Define what you expect

–Don't make things too complex at first

–Tell your agents!

●Plan a call review process

–Don't oversample issues!

●Define initial QA review targets

●Plan in advance for corrective action to be taken

QA – Getting started

A simple form to review issues

●Track whether the problem is solved

●Track the sex of the caller

●If problem not solved…

–Track conversation metrics

●Different data input

●Non-scoring questions

●Show-hide questions

QA – A real-life sample

Start by defining questions...

●Engagement code

●Description

●Weight

●Shortcuts

●Value types

–Number

–Yes/No

–Combo (drop-down)

QA – Creating items

Then create forms...

●Security keys

●Allowed queues

●Open / Locked

●Sections

●Thresholds

QA – Creating forms

Define which items are a part of the form.

QA – Creating forms (2)

●Group items into

sections

●Move up and down

●Score contributions

●Visibility

Once a form has items in it, it cannot be changed.

●Form selector

●Call information

●Recordings

●Timer

●Items per type

●Overview per section

●Score

●Questions & notes on questions

Anatomy of a form

What does a form look like?

●General notes

●Notes per question

Audio player

●Speed controls

●Playback controls

●Markers

Anatomy of a form (2)

What does a form look like? (continued)

Run Report → Queue details → Call Detail → QA

Form appears → Listen to the call → Fill it.

QA InputEntering QA is really simple.

Results:

●Per agent

●Per queue

●Per group

●Per analyst

Showing:

●Averages

●Items by threshold

QA ResultsTo see results: Run QA reports → Criteria

Divided by section:

●By agent:

–Calls

–Average per question

●By queue:

–Calls

–Average per question

QA Results (2)More results...

See all the graded calls that contribuited to a result.

Click on a call to see its QA form (questions, notes, associated audio files).

QA Results (3)

How did a result came to be?

●Overall summary

●Scoring questions

●N. forms scored

–Averages

–% per threshold

–% shortcuts

●Non scoring

–Averages

–Histograms

QA Results - SummaryOverview of results

Contents

●Introduction

Open discussion

Deciding which calls to grade

Just clicking on random calls is not especially effective.

●Poor ergonomics

●Selection biases:

–Issues / „interesting“ calls

–Short / long calls

–First / last calls

Solution:

●Create a policy

–By call outcome

–By agent group

●Use the Grader's page

–Weighted random sampling

QA – Grading calls

Find efficent sets of calls to be graded

●Random sampling in call space

●Prefers calls matching multiple criteria

●Multiple Quality Analysts can work on

the same issue at once

●Shows target statistics

●First line of defence against quality

issues

–Ask for coaching

–Ask for CBTs to be taken

QA – Effective grading

Contents

●Introduction

Open discussion

Long-term agent performance management

●Agents need to be managed in order to work effectively

–Performance targets

–Quality policies

–Agents need training

●Agent lifecycle

–Training

–Validation period

–Review

–Production

–Periodic Review

QA – Agent Performance

Comprehensive Performance Rules

●Performance

–Sales, SPH

–Contacts

●Call duration

–Averages

–Anomalies

●Quality results

Red and yellow alarms with relative importance scores.

QA – Agent performance

The Performance tracker lets you:

●Apply Performance rules and find anomalies

●Send training and coaching

●Move agents between groups

●Be reminded for further review

QA – Performance tracker

Contents

●Introduction

–Automated QA input

–Queue recalls with WombatDialer

–Handling VIP callers

●Open discussion

Contents

●Introduction

●Open discussion

QA is an effective process, but...

–Reviewing calls manually is very expensive

–Most calls are not very informative

–Biased sampling

–What if we could automate it?

●How can this work?

–Agent tranfers call to IVR at the end of the interaction – 70-80% of callers will accept

–IVR gathers simple information („Are you satisfied?“ - „Problem solved?“)

–Asterisk pushes this information to QM as a QA Form

●Targets:

–You have a way to review issues immediately

–You can monitor quality fairly and continuously

Surveys: Automating QA

●Create a QA form called „SATISF“

–One section

–One single yes/no question - „Are you satisfied with the interaction?“ (item SAT)

●Create a remote user in QM

–User „qasubmit“ password „passw0rd“

–Class ROBOTS

–Custom security key „QATRACK“

Surveys – QM set-up

Asterisk issue: IVR happen on a different channel:

●When calling the queue, store the original UniqueId of the call and

the Queue used on inheritable channel variables

●Allow unattended transfers on the queue (# + xxx)

●Download script pushQA.sh from the Open QueueMetrics Add-

ons on GitHub

●Make it executable and edits its configuration

●Create a simple IVR script to transfer to

Surveys – Asterisk set-up

exten => _999,1,Answer

exten => _999,n,Background(ivr_satisfaction)

exten => _999,n,Wait(10)

exten => _999,n,Hangup()

; Not happy

exten => 0,1,Playback(beep)

exten => 0,n,System(/var/lib/asterisk/agi-bin/pushQA.sh ${UID} ${QUE} 0)

exten => 0,n,Playback(thank-you)

exten => 0,n,Hangup

; Happy

exten => 1,1,Playback(beep)

exten => 1,n,System(/var/lib/asterisk/agi-bin/pushQA.sh ${UID} ${QUE} 100)

exten => 1,n,Playback(thank-you)

exten => 1,n,Hangup

Surveys – Asterisk set-up

exten => _900,1,Wait(1)

exten => _900,n,Answer

exten => _900,n,Playback(queue-welcome)

exten => _900,n,Set(CHANNEL(musicclass)=mymusic)

exten => _900,n,Set(__QUE=myqueue)

exten => _900,n,Set(__UID=${UNIQUEID})

exten => _900,n,Queue(${QUE},t,,,300)

exten => _900,n,Hangup

Asterisk dialplan

Push QA to QueueMetrics

Store Queue & UID

Contents

●Introduction

●Open discussion

Your services have a known peak time.

●You need to staff agents based on the load at peak time

●Your wait time SLA is stressed at peak time

●Harder interaction with callers who have waited too long

Solution: offer people the choice to be recalled

●Better SLAs – less staff required

●Happier customers

●Can be extended to web-based „call me back“

Automated recalls

Caller is offered the option to press '1' to be recalled

●If they do, a script starts to get their phone number

●When done, the number is piped to WombatDialer

WombatDialer monitors the recall queue for free agents

●When agents are available on it, recalls starts

●Wombat knows how to handle busy, no answers, etc.

When the call is started...

●A script plays a welcome message on connection

●The call is queued on the Recalls queue

Auto recalls: the logic

QueueMetrics

●Situational awareness

●Agent management

Asterisk

●Script to track caller's numbers

●Script to greet people being recalled

WombatDialer

●Monitors presence on Asterisk queues

●Implements recalls

●Handles busy, no answers, invalid numbers, etc.

Auto recalls: components

QueueMetrics configuration

●Create two queues – one for inbound and one for recalls

●Monitor them through the Real-Time page

●Supervisor moves agents to and from the Recalls queue based on

the state of the Inbound queue

●Everything else happens automatically

Auto recalls: QM

WombatDialer

●Runs multiple independent campaigns in the background

●Receives numbers to be called and extra information for handling

them over HTTP

●Monitors agent continuously

–Are they available? Paused? Busy?

●Handles recalls with sensible rules

–Retry in 10 minutes if you get a busy

–Retry in 30 minutes if the number is free

–Call only during allowed times

●When successful, connects to the welcome script

Auto recalls: WombatDialer

●Solution can be implemented in very little time

●Does not require extensive changes to the PBX config

●Use the same tools you use for inbound

●Very extensible solution

–Reverse IVRs

–Voice synthesis

●Excellent ROI

Recalls – all together

Contents

●Introduction

●Open discussion

How can we handle VIP callers?

●Special low-latency queues

●Different products

●Automatic routing of open cases

●How can we identify callers?

Usually done with dedicated DIDs, but...

●Hard to scale to hundreds or thousands of cases

●We can definitely do better!

VIP callers

VIP callers in QM

QueueMetrics offers „Known numbers“, an easy-to -query

database of known numbers.

●Associates an action and an identity

to the caller

●Action can be time-limited

●Optional „agent affinity“ to first try the

agent who is currently handling the

●Database can be fed through JSON

VIP callers in QM (2)

For security reasons, you need a special user to acces

QueueMetrics through the APIs.

●Create a remote user in QM

–User „pbxapi“ password „api123“

–Class ROBOTS

–Custom security key „PBXAPI“

VIP callers in AsteriskYou query QueueMetrics before routing a call:

same => n,Set(CURLOPT(hashcompat)=yes)

same => n,Set(NUM=${CALLERID(number)})

same => n,Set(URL=http://my.qm/queuemetrics/numberLookup.do

?mode=hash&user=pbxapi:api123))

same => n,Set(HASH(resp)=${CURL(${URL}&number=${NUM})})

same => n,Set(CALLERID(name)={HASH(res,name})

same => n,GotoIf($["${HASH(resp,action)}" = "VIP"]?supervip)

same => n,GotoIf($["${HASH(resp,action)}" = "BLACKLIST"]?blklist)

same => ....

So with one single call you get:

●Name of the caller

●VIP or Blacklist

●Agent affinity

And you can route accordingly!

VIP callers in AsteriskYou query QueueMetrics before routing a call:

same => n,Set(CURLOPT(hashcompat)=yes)

same => n,Set(NUM=${CALLERID(number)})

same => n,Set(URL=http://my.qm/queuemetrics/numberLookup.do

?mode=hash&user=pbxapi:api123))

same => n,Set(HASH(resp)=${CURL(${URL}&number=${NUM})})

same => n,Set(CALLERID(name)={HASH(res,name})

same => n,GotoIf($["${HASH(resp,action)}" = "VIP"]?supervip)

same => n,GotoIf($["${HASH(resp,action)}" = "BLACKLIST"]?blklist)

same => ....

So with one single call you get:

●Name of the caller

●VIP or Blacklist

●Agent affinity

And you can route accordingly!

Contents

●Introduction

●Open discussion

Thank you for attending!

QueueMetrics www.queuemetrics.com

Loway www.loway.ch

A real programmer puts two glasses on his bedside table before going to sleep.A full one, in case he gets thirsty, and an empty one, in case he doesn’t.

Contact us

pinterest.com/QueueMetrics

twitter.com/queuemetrics

linkedin.com/company/loway

slideshare.net/QueueMetrics

youtube.com/user/QueueMetrics

plus.google.com/+QueuemetricsCallCenter

facebook.com/QueueMetrics

loway.ch/rss.jsp

QueueMetrics - Tips and Tricks

Technology