Improving the IP Telephony Experience: How to Troubleshoot Converged Networks with VoIP monitoring...

www.wildpackets.com © WildPackets, Inc.

www.wildpackets.com © WildPackets, Inc. WildPackets Seminar Series

© WildPackets, Inc.

Corporate Background

• Experts in network monitoring, analysis, and troubleshooting

Founded: 1990 / Headquarters: Walnut Creek, CA

Offices throughout the US, EMEA, and APAC

• Our customers are leading edge organizations

Mid-market, and enterprise lines of business

Financial, manufacturing, ISPs, major federal agencies,

state and local governments, and universities

Over 7,000 customers / 60+ countries / 80% of Fortune 1,000

• Award-winning solutions that improve network performance

Internet Telephony, Network Magazine, Network Computing Awards

United States Patent 5,787,253 issued July 28, 1998 • Different approach to maintaining availability of network services

WildPackets Seminar Series 3


What We Do

• Provide network visibility and intelligence … WatchPoint, OmniPeek, OmniEngines

• Expert systems – we find the problems for you

• Superior drill-down capability – trouble-shoot from anywhere

• Flexible, customizable, extensible – leverage your investment

Professional services, training, best practices

• For all network segments … Data center to desktop to remote office

LAN, WAN, Wireless …

HTTP, Email, Database, VoIP, Video …

• To … Network engineers; IT Management; Developers



With accurate visibility

into the network…

IT staff can improve:

• End-user Productivity

• Network Performance

• Application Performance

• Security

• Compliance



Select WildPackets Customers


Mid-Market / Enterprise Government & Education


WildPackets Delivers Network Visibility




Feel the Rush…

• Your network is

running great!

• Packets enjoy a

speed-limit ride

on the wire!

• Performance is

awesome!

• You have few

complaints from

users!

Are You Dreaming?


Or Feel the Jam!

• Does your network really look more like this…?

CAMP IT Pinpointing the Problem 10


The BIG Questions

• DID YOU (OR WILL YOU) DETERMINE THE HEALTH

OF YOU NETWORK BEFORE DEPLOYING VOIP?

• DID YOU (OR WILL YOU) RID YOUR NETWORK OF

VoIP-KILLING TROUBLES BEFORE INSTALLATION?

• CAN YOU SUCCESSFULLY ASSESS YOUR

NETWORK WHEN VOIP-KILLING TROUBLES ARISE?


Troubleshooting?

• The formal definition of troubleshooting is… “the act of shooting or killing troubles”


When troubles are small, they can

seem so innocent and harmless,

but…


Troubleshooting?



You’ve got to kill them when they’re

young, or…


Troubleshooting?



They will come

back to get you!


Troubleshooting: Not Just Reactive!

• When we use the word troubleshooting, most folks

immediately think about reacting to a problem

• But proactive troubleshooting identifies troubles when they are small and are having minimal impact!

• The concept is simple…

Proactive

Troubleshooting

Reactive

Troubleshooting =


What Troubles Are We Shooting for VoIP?

• Before VoIP, network

troubleshooting focused on

factors like application

response time and latency

• With VoIP, we’ve

learned that latency

is just one part of a

three-headed

monster…


Jitter Latency

Packet

Loss

The monster attacks RTP with one or more of its weapons!


Identifying Troubles: The First Step

• Before you begin worry about statistics or packets,

take time to listen to representative calls

• Hearing VoIP troubles is the most natural way to recognize them

Use analysis application that can playback call audio • Playback of individual RTP streams

• Playback of complete call

Listen for the telltale of signs of latency, jitter, and packet loss


Understanding the Monster: Latency

• The time it takes for packets to travel across the

network is based on several factors: Distance latency – unavoidable – a fact of physics

Queue latency

Decision latency

Encryption/decryption

Codec operations



Queue Latency &

Decision Latency

Network

Propagation

Delay

Encoding / Decoding

Compression / Decompression

Jitter Buffer Latency


Latency Tolerance

0 ms

100 ms

200 ms

300 ms

400 ms

500 ms

600 ms

700 ms

800 ms

High Quality

Required for

VoIP

Satellite

Quality

Fax Relay

Broadcast

Quality

The ITU

recommends a

maximum one-way

delay of 150 ms

for VoIP

Roundtrip latency > 250 ms will be

noticeable for call participants!

Latency is much

more critical for

VoIP systems than for

traditional data

applications



• Excessive latency is a major enemy of VoIP Often caused by network congestion in the absence of adequate

QoS provisions

For some network segments, especially WAN circuits, elevated

latency may be a way of life

Excessive latency may be one-way or roundtrip, depending on

how traffic is routed through the network


Latency's Effects: Talkover

• “Talkover” occurs when excessive latency delays

audio

– Caller A speaks, but his words are slow to reach Caller B

– As a result, Caller B is slow to respond

– Caller A believes that his words were unheard, so he begins to

speak again, often just as Caller B begins his response

– Caller A is speaking as he begins to hear Caller B

– Caller B may still be speaking when Caller A’s second set of

words begin to arrive

• Conversation cadence is not natural or comfortable

• Callers feel as if they must “push to talk” or say

“over” to control the conversation


Latency's Effects: Echo

• In some cases, excessive latency may produce an

echo effect The speaker’s voice feeds back into the listener’s microphone

The speaker then hears his own voice returning from the

listener’s end, but delayed due to latency

• Most callers find it difficult to maintain normal speech when echo delay is prolonged

• Some VoIP systems attempt to cancel echo, but are

not always successful

High latency may also cause additional troubles such as loss of

synchronization between audio and video for multimedia sessions.


Understanding the Monster: Jitter

• Closely related to latency Jitter is really nothing more than variable latency preventing on-

time delivery of RTP packets

Example • G.711 needs an RTP packet to be delivered every 20 ms to provide

accurate audio reconstruction

• If the delta time between one RTP packet and the next is 24 ms,

then the jitter is 4 ms

• VoIP devices employ jitter buffers to smooth packet

delivery Jitter up to about 100 ms may be managed by the buffer

Packets with jitter greater than the jitter buffer are dropped

An large jitter buffer increases latency


Jitter's Effects

• Jitter causes weird “sound effects” that vary with

jitter severity and environmental factors

• Examples include: Static

Stuttering or uneven audio – abnormal speech rhythm

For multimedia systems, video may be “jerky” or irregular

• If jitter levels are high, packet loss can result In some cases, severe jitter may sound similar to packet loss,

even if no packets are actually dropped


Measuring Jitter

• Jitter can be measured as instantaneous jitter The difference in actual packet arrival time vs. expected arrival time

• Each packet is the jitter reference for the next packet

Gives a rather “jerky” view of jitter that may overstate its effects • However, it does correctly depict the “jerkiness” of a call

• Smoothed jitter is an improved metric defined in RFC 3550 Applies a “filter” to smooth out the instantaneous jitter trend, which

provides a much more useful and accurate view of jitter over time

For smoothed and instantaneous jitter, minimum, maximum, average,

and standard deviation values are very meaningful.

Graphs of these metrics provide good insight into call quality.


Measuring Jitter

• Absolute jitter uses the first packet in the stream as a

constant reference, which catches “clock skew” Clock skew is the difference in the clocks on the involved VoIP

stations and the measuring device

Absolute jitter graphs can reveal when packets are dropped

due to clock skew between VoIP stations

Calculating maximum, minimum, average, and standard deviation for absolute

jitter is not very helpful because of the clock skew factor.


Understanding the Monster: Packet Loss

• Most commonly caused by: Packet dropped due to physical layer corruption

Congestion without adequate QoS provisions

Jitter buffer discards due to excessive latency

• Causes missing sounds, syllables, words, or phrases DSP algorithms may compensate for up to 30 ms of missing data

More than 30 ms of missing audio (e.g. 2 RTP packets for G.

711) is noticeable by listeners


Packet Loss Effects

• An average person speaks at a rate of about 200

words per minute Do the math – that’s 3.33 words/sec = 300 ms per word

For G.711, we would need to lose 15 consecutive RTP packets

to lose a whole word

Dropping 15 packets/sec for G.711 would be a loss rate of 30%

• But losing only a few packets can still be very noticeable

Loss of more than 2 consecutive packets will be heard

Loss rates 2% will have a strong impact on quality

Losses of 5 – 10% make calls all but intolerable

Bursty periods of packet loss are worse than more dispersed

loss


Packet Loss Bursts

• A packet loss “burst” is a period of time that begins

and ends with loss in which the number of

consecutive received packets is less than the minimum number needed (G

min) to maintain adequate

quality G

min for VoIP = 16

Gmin

for video services 64 - 128

• The more “bursty” the packet loss, the worse the

quality of the call

• VoIP quality scoring standards consider Burst length (ms) of the bursts for a given RTP stream

Burst density (% of missing packets) for the RTP stream


Assessing the Monster's Impact

• While traditional network applications are very tolerant of

jitter, latency, and even some degree of packet loss, VoIP is

very sensitive to these troubles

• Levels of jitter, latency, and packet loss that would be easily

tolerated on a data network can be devastating on a

converged VoIP network

• Pre- and post-deployment network assessment are critical

– You must understand your network’s ability to accommodate VoIP

– Current latency, jitter, and packet loss

– QoS capabilities

– Current bandwidth utilization (is there any room for VoIP)

– You must maintain a constant vigil after deployment to watch for

imminent troubles


Network Traffic: Quantitative Analysis

• Most network folks are concerned about the amount

of traffic on their networks Utilization (percentage of bandwidth)

Throughput (bits or bytes per second)

• You also need to be concerned about individual utilization components

How much bandwidth and throughput can be attributed to each

application or process? • Clarifies which application traffic may need to be tuned or controlled

How well or poorly will the baseline (trended) behavior of each

application interact with VoIP • Don’t forget to also consider the reverse case – VoIP’s impact on

existing applications


The Impact of "Just One More Call"

• Although a network link may be able to support a

number of concurrent calls, one additional call is

often enough to cause quality problems…

x1113

x2111

x1112

x1111

1st Call

2nd Call

3rd Callx2112

x2113

Example: The WAN can support 2 simultaneous calls.

What happens when a third call is attempted???


Network Traffic: Qualitative Analysis

• The quality of your network traffic is potentially more

important than its quantity when it comes to VoIP

• Many traffic streams are “bursty” in nature Burstiness my occur over long period of time, or may consist of

rapid, recurring traffic spikes

Prolonged rises in utilization may decrease the number of calls

that can occur simultaneously

Sharp spikes may cause very noticeable quality issues with

ongoing calls

• Your baseline monitoring should consider not only

averages and long-term trends, but also the short-

term peaks and dips that characterize your traffic

flow


Troubleshooting for Latency,

Jitter, and Packet Loss • Proactive and reactive assessment of VoIP troubles

can be easily accomplished using common utilities

and/or analysis tools: Performance and protocol analysis applications

Network management systems

PING, Traceroute, etc. • Use packet sizes that reflect your actual RTP packets

• For example, 218 byte RTP packets are typical for G.711

For reactive assessment (post-deployment) include VoIP stats,

quality scores, and audio evaluation

© WildPackets, Inc. 36

ping –l 218 –t 206.169.32.70

Pinging 206.169.32.70 with 218 bytes of data:

Reply from 206.169.32.70: bytes=218 time=88ms TTL=57 Reply from 206.169.32.70: bytes=218 time=83ms TTL=57 Reply from 206.169.32.70: bytes=218 time=92ms TTL=57 Reply from 206.169.32.70: bytes=218 time=88ms TTL=57 Reply from 206.169.32.70: bytes=218 time=88ms TTL=57 Reply from 206.169.32.70: bytes=218 time=102ms TTL=57 Reply from 206.169.32.70: bytes=218 time=87ms TTL=57

Watch for

excessive

latency.

Watch for

unsteady

latency (jitter).

Watch for

variable

routing.

Watch for

packet

loss.

Ping statistics for 206.169.32.70: Packets: Sent = 7, Received = 7, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 83ms, Maximum = 102ms, Average = 90ms

C:\>


Traceroute can show the hop-by-

hop components of latency to help

locate the source of troubles.

C:\>tracert 206.169.32.70

Tracing route to 206.169.32.70 over a maximum of 30 hops:

1 1 ms <1 ms <1 ms 10.71.0.1 2 3 ms 4 ms 3 ms 69.46.168.254 3 3 ms 2 ms 2 ms 172.33.37.78 4 4 ms 3 ms 4 ms 66.163.28.157 5 6 ms 2 ms 2 ms 209.167.101.217 6 8 ms 3 ms 2 ms 152.63.133.74 7 15 ms 13 ms 14 ms 152.63.128.117 8 14 ms 14 ms 14 ms 152.63.66.65 9 17 ms 102 ms 68 ms 64.215.195.137 10 17 ms 14 ms 14 ms 67.17.109.118 11 14 ms 15 ms 14 ms 207.200.10.11 12 73 ms 73 ms 72 ms 66.192.254.213 13 87 ms 83 ms 107 ms 206.169.32.70

Trace complete.

C:\>


A Picture is Worth a 1000 Words

• A graph of latency,

jitter, or packet loss

can speak volumes

about network

health, either for

proactive or reactive

troubleshooting

• Overlaying this

graph with a graph

of utilization or total

throughput can

reveal even more

about the causes of

VoIP troubles


Quality Scoring for VoIP

• One of the best initial troubleshooting tools for VoIP traffic

• Mean Opinion Score (MOS) – several flavors Algorithmic simulation of subjective audio assessment Most commonly used varieties are MOS-LQ (listening quality) and MOS-CQ (conversational quality) Possible range of 1 (poor) to 5 (excellent) Maximum possible MOS = 4.4 with G.711 Typical range in most networks is 3.5 – 4.2

• R-Factor – several flavors Based on latency, jitter, packet loss, bit rate, and signal-to-noise ratio, codec effects (for low bit-rate codecs), recency

• The ITU algorithms consider about 20 quality inputs

Possible range of 0 (poor) to 100 (excellent) Provides LQ, CQ, and other score variants


Quality Score Trending

• Isolated scores are useful for validating single call

complaints, but overall VoIP health is best seen by

graphing long-term trends

Overlaying VoIP trends

with network utilization,

errors, or other metrics may

reveal previously unseen

performance relationships!


Got QoS?

• One of the most potent weapons for fighting VoIP troubles is provision of Quality of Service (QoS) parameters

• QoS enables network devices to prioritize and give preference to packet streams that are sensitive to delay, packet loss, jitter, and other performance inhibitors

• Standards-based QoS methods include: RSVP (nearly antiquated) IP Differentiated Services (DiffServ) MAC Layer QoS with IEEE 802.1p VLANs

• QoS may be obtained or supplemented via proprietary means, such as traffic shaping via various flow processing algorithms


QoS the Old-Fashioned Way with RSVP

RSVP is fading away since it manages traffic on a flow-by-flow basis, a method

that is not scalable to enterprise and carrier grade networks.


IP Differentiated Services

• Using DiffServ, routers and other devices no longer have to worry about individual flows that are identified by IP addresses and port numbers

• Instead, devices only need to examine 6 bits to know how to classify and manage traffic

The DiffServ bits provide 64 DSCPs, of which 8 give

backward compatibility with the older IP Type of Service

(ToS) field. Only 32 DSCPs are now in common use.

DiffServ Code

Point (DSCP)


DiffServ at the Data Link Layer

• IEEE’s 802.1p specification enables packet prioritization via

3-bit field in 802.1q VLAN tags

• This field provides 8 levels of precedence, and only requires

switches to read 3 bits to classify and manage traffic

• Each 802.1p-aware switch allocates 8 different queues to

separate handling for each priority level

• Network administrators must map priority levels with handling

methods

VLAN Tag (802.1q)


Ready for QoS?

• QoS provisions are based on the “weakest link”

concept If any device in a data path does not support QoS, then media

streams will not be afforded the preference they require for good

performance

• Pre-deployment assessment must ensure that ALL devices can recognize and respond to QoS

parameters in packet headers Switches, routers, firewalls, proxies, and any other devices that

touch RTP packets must be “VoIP-friendly”


Summary

• The main threats to VoIP are latency, jitter, and packet loss

The presence of these “monsters” may remain unnoticed in a data network, but will become very obvious and ugly in VoIP systems

• Troubleshooting VoIP is both reactive and proactive Prudence dictates that you test your network before installing a VoIP system to identify and correct performance troubles

• You can see the three-headed monster even before VoIP is installed

After VoIP deployment, constant monitoring will • Validate QoS operations

• Reveal network traffic pattern changes that adversely affect VoIP • Provide alerts when VoIP performance declines

• Proactive troubleshooting and monitoring is a way of life – a job that is never done!

www.wildpackets.com © WildPackets, Inc. WildPackets Seminar Series


OmniPeek Enterprise Packet Capture, Decode and Analysis

• 10/100/1000 Ethernet, Wireless, WAN, 10Gbe

• Portable Capture and OmniEngine Console

• VoIP Analysis and Call Playback

OmniEngine / Omnipliance Distributed Enterprise Network Forensics

• Packet Capture and real-time analysis

• Stream-to-Disk with Data Mining

• Integrated OmniAdapter network analysis cards

Product Line Overview



OmniPeek Product Family Architecture

• OmniEngines / Omnipliance OmniEngines collect and process

data throughout the network.

• Platform Services Filters, alarms, authentication and

communication.

• Interfaces Packets captured from Ethernet

NICs, 802.11 NICs, OmniAdapter

Gigabit and WAN analyzer cards.

• OmniPeek Analyzers OmniPeek Analyzers perform local

network analysis on a portable basis and function as consoles to

multiple remote engines.

• Intelligent Data Transport OmniPeek analyzes network traffic

at the engine and intelligently

coordinate analysis results that are exchanged with the console for

maximum efficiency.



Omnipliance

• Full Line Rate support, Windows or Linux

• Customer upgradable hardware

• Representative Specs 3U, 2x Intel Dual Xeon 3.3 Ghz,

8x 500 Gigabyte SATA or 8x 150 Gigabyte SAS, etc.

Packet filtering in hardware

1U Omnipliance Edge available



WatchPoint Architecture

• WatchPoint Server

Web server

Centralized data collector management

• OmniFlow Collector

• NetFlow Collector

• sFlow Collector

• Scalable architecture

• Collectors can be separated for

increased performance

• Published API and SDK

for easy extension



WildPackets Key Differentiators

• Visual Expert Intelligence with Intuitive Drill-down

Let computer do the hard work, and return results, real-time

Packet / Payload Visualizers are faster than packet-per-packet diagnostics

Experts and analytics can be memorized and automated

• Automated Capture Analytics

Filters, triggers, scripting and advanced alarming system combine to provide

automated network problem detection 24x7

• Multiple Issue Network Forensics

Can be tracked by one or more people simultaneously

Real-time or post capture

• User-Extensible Platform

Plug-in architecture and SDK

• Aggregated Network Views and Reporting

NetFlow, sFlow, and OmniFlow



WildPackets, Inc.

1340 Treat Boulevard, Suite 500

Walnut Creek, CA 94597

(925) 937-3200

Date post:	20-May-2015
Category:	Technology
Upload:	wildpackets
View:	2,894 times
Download:	4 times

Improving the IP Telephony Experience: How to Troubleshoot Converged Networks with VoIP monitoring...

Technology