Date post: | 04-Jun-2018 |
Category: |
Documents |
Upload: | bhupinder-rajpoot |
View: | 273 times |
Download: | 10 times |
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 1/70
LTE E2E Field Network Performance -Definitions of Key Performance Indicators
DN0951948
Issue 01 DRAFT
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 2/70
2
DN0951948 Issue 01 DRAFT
LTE E2E Field Network Performance - Definitions of Key Performance Indicators
Id:0900d80580729625
The information in this document is subject to change without notice and describes only the
product defined in the introduction of this documentation. This documentation is intended for the
use of Nokia Siemens Networks customers only for the purposes of the agreement under whichthe document is submitted, and no part of it may be used, reproduced, modified or transmitted
in any form or means without the prior written permission of Nokia Siemens Networks. The
documentation has been prepared to be used by professional and properly trained personnel,
and the customer assumes full responsibility when using it. Nokia Siemens Networks welcomes
customer comments as part of the process of continuous development and improvement of the
documentation.
The information or statements given in this documentation concerning the suitability, capacity,
or performance of the mentioned hardware or software products are given "as is" and all liability
arising in connection with such hardware or software products shall be defined conclusively and
finally in a separate agreement between Nokia Siemens Networks and the customer. However,
Nokia Siemens Networks has made all reasonable efforts to ensure that the instructions
contained in the document are adequate and free of material errors and omissions. Nokia
Siemens Networks will, if deemed necessary by Nokia Siemens Networks, explain issues which
may not be covered by the document.
Nokia Siemens Networks will correct errors in this documentation as soon as possible. IN NO
EVENT WILL Nokia Siemens Networks BE LIABLE FOR ERRORS IN THIS DOCUMENTA-
TION OR FOR ANY DAMAGES, INCLUDING BUT NOT LIMITED TO SPECIAL, DIRECT, INDI-
RECT, INCIDENTAL OR CONSEQUENTIAL OR ANY LOSSES, SUCH AS BUT NOT LIMITED
TO LOSS OF PROFIT, REVENUE, BUSINESS INTERRUPTION, BUSINESS OPPORTUNITY
OR DATA,THAT MAY ARISE FROM THE USE OF THIS DOCUMENT OR THE INFORMATION
IN IT.
This documentation and the product it describes are considered protected by copyrights and
other intellectual property rights according to the applicable laws.
The wave logo is a trademark of Nokia Siemens Networks Oy. Nokia is a registered trademark
of Nokia Corporation. Siemens is a registered trademark of Siemens AG.
Other product names mentioned in this document may be trademarks of their respectiveowners, and they are mentioned for identification purposes only.
Copyright © Nokia Siemens Networks 2010. All rights reserved
f Important Notice on Product SafetyElevated voltages are inevitably present at specific points in this electrical equipment.
Some of the parts may also have elevated operating temperatures.
Non-observance of these conditions and the safety instructions can result in personal
injury or in property damage.
Therefore, only trained and qualified personnel may install and maintain the system.
The system complies with the standard EN 60950 / IEC 60950. All equipment connected
has to comply with the applicable safety standards.
The same text in German:
Wichtiger Hinweis zur Produktsicherheit
In elektrischen Anlagen stehen zwangsläufig bestimmte Teile der Geräte unter Span-
nung. Einige Teile können auch eine hohe Betriebstemperatur aufweisen.
Eine Nichtbeachtung dieser Situation und der Warnungshinweise kann zu Körperverlet-
zungen und Sachschäden führen.
Deshalb wird vorausgesetzt, dass nur geschultes und qualifiziertes Personal die
Anlagen installiert und wartet.
Das System entspricht den Anforderungen der EN 60950 / IEC 60950. Angeschlossene
Geräte müssen die zutreffenden Sicherheitsbestimmungen erfüllen.
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 3/70
DN0951948 Issue 01 DRAFT
3
LTE E2E Field Network Performance - Definitions ofKey Performance Indicators
Id:0900d80580729625
Table of ContentsThis document has 70 pages.
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2 Measurement Methods and Reference Conditions . . . . . . . . . . . . . . . . . 9
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.2 Reference Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.2.1 Distinguished Cell Positions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.2.2 Channel Models. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.2.3 Template for the Description of Reference Conditions . . . . . . . . . . . . . 12
2.3 Measurement Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.3.1 Throughput Measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.3.1.1 Cell Throughput . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.3.1.2 Peak User Data Rate. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172.3.2 Latency Measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.3.2.1 Round Trip Time Measurement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.3.2.2 Signaling and Media Delay Measurements . . . . . . . . . . . . . . . . . . . . . . 19
2.3.2.3 Reference Conditions for Latency Measurements. . . . . . . . . . . . . . . . . 20
2.3.3 Service Accessibility Measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.3.4 Reliability Measurements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.3.4.1 Transport Error Rates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.3.4.2 Service Drop Rates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.3.5 Mobility (Handover) Measurements. . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.4 Measurement Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.4.1 Field Network Cluster. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.4.2 Trial Lab. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.4.3 Measurement Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.4.3.1 Subscriber and Equipment Traces. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.4.3.2 PM Counters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.5 Measurement Reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3 Definitions of Key Performance Indicators (KPIs) . . . . . . . . . . . . . . . . . 32
3.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.1.1 Application Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.1.2 LTE Network Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.1.2.1 LTE E2E Network Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333.1.2.2 Radio Bearer Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.2 KPI Definition Template . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3.3 Application Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.3.1 Data Download / Upload (FTP) Service KPIs . . . . . . . . . . . . . . . . . . . . 36
3.3.1.1 (FTP) Service Accessibility Ratio. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.3.1.2 (FTP) Completed Session Ratio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.3.1.3 (FTP) Service Access Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.3.1.4 (FTP) User Data Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.4 LTE Network Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.4.1 LTE E2E Network KPIs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.4.1.1 Attach Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 4/70
4
DN0951948 Issue 01 DRAFT
LTE E2E Field Network Performance - Definitions of Key Performance Indicators
Id:0900d80580729625
3.4.1.2 Attach Success Rate. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.4.1.3 Service Request Time (EPS), UE Initiated . . . . . . . . . . . . . . . . . . . . . . . 45
3.4.1.4 Service Request (EPS) Time, Network Initiated . . . . . . . . . . . . . . . . . . . 49
3.4.1.5 Service Request (EPS) Success Rate . . . . . . . . . . . . . . . . . . . . . . . . . . 503.4.1.6 Service (EPS Bearer) Drop Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
3.4.1.7 Handover Procedure Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
3.4.1.8 Handover Success Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
3.4.1.9 Paging Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
3.4.1.10 Paging Failure Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
3.4.1.11 (LTE) Round Trip Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
3.4.1.12 (LTE) User Data Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
3.4.1.13 (LTE) Packet Loss Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
3.4.2 Radio Bearer KPIs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
3.4.2.1 (RB) Packet Loss Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
3.4.2.2 (RB) User Data Rate. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 653.4.2.3 Cell Throughput . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
3.4.2.4 (RB) Service Interrupt Time (HO) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
4 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 5/70
DN0951948 Issue 01 DRAFT
5
LTE E2E Field Network Performance - Definitions ofKey Performance Indicators
Id:0900d80580729625
List of FiguresFigure 1 LTE Bearer Service Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Figure 2 Distribution of Users for Cell Throughput Measurements . . . . . . . . . . . 15
Figure 3 Reference Cluster for Field Performance Verification Tests . . . . . . . . . 28
Figure 4 KPIs for Packet Switched Data Services . . . . . . . . . . . . . . . . . . . . . . . . 38
Figure 5 Attach Procedure w/ Initial EPS Bearer Establishment . . . . . . . . . . . . . 42
Figure 6 EPS Bearer Setup Procedure, Part 1 . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Figure 7 EPS Bearer Setup Procedure, Part 2 . . . . . . . . . . . . . . . . . . . . . . . . . . 48
Figure 8 Handover Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
Figure 9 Paging Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 6/70
6
DN0951948 Issue 01 DRAFT
LTE E2E Field Network Performance - Definitions of Key Performance Indicators
Id:0900d80580729625
List of TablesTable 1 Channel Models and Doppler Frequencies . . . . . . . . . . . . . . . . . . . . . . 11
Table 2 Description of Reference Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Table 3 KPI Categories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Table 4 Reference Conditions for Cell Throughput Measurements . . . . . . . . . . 16
Table 5 Reference Conditions for User Data Rate Measurements . . . . . . . . . . . 18
Table 6 Reference Conditions for Latency Measurements . . . . . . . . . . . . . . . . . 21
Table 7 Reference Conditions for Success Rate Measurements . . . . . . . . . . . . 22
Table 8 Reference Conditions for Packet Loss Rate Measurements . . . . . . . . . 24
Table 9 Reference Conditions for Service Drop Rate Measurements . . . . . . . . 26
Table 10 Reference Conditions for Handover Measurements . . . . . . . . . . . . . . . 27
Table 11 Overview of Application Service KPIs . . . . . . . . . . . . . . . . . . . . . . . . . . 33
Table 12 Overview of LTE E2E Network Service KPIs . . . . . . . . . . . . . . . . . . . . . 33
Table 13 Overview of LTE Radio Bearer Service KPIs . . . . . . . . . . . . . . . . . . . . 34Table 14 KPI Definition Template . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 7/70
DN0951948 Issue 01 DRAFT
7
LTE E2E Field Network Performance - Definitions ofKey Performance Indicators
Introduction
Id:0900d8058070a69b
1 IntroductionThe current document provides the following information:
• Definitions of End-to-End Key Performance Indicators (E2E KPIs) for Long-TermEvolution (LTE)
• Descriptions of measurement methods for field and lab trials
KPIs are basic network quality indicators for offer processing and field acceptance tests.
The KPIs specified in this document are E2E related KPIs; they help network operators
and Nokia Siemens Networks to determine the quality of deployed LTE networks.
In order to prove that the performance of an operating network matches the targeted
quality, field acceptance tests are carried out. As multiple factors may influence the per-
formance of an operational network, detailed conditions, under which network KPIs are
valid, must be specified and agreed upon. These include system performance, network
planning and dimensioning, measurement campaign planning and parameterization.
These conditions will be reffered to as "reference conditions" in the sequel.
LTE system architecture was designed to meet the following major goals. These are
also the constant focus of lab and field performance tests (for more details, see Refer-
ences 7. [3GPP25.913]):
• Improved system capacity & coverage
• High user data rates (peak upload and download rates)
• Reduced latency (one-way delay and RTT)
• Simplified architecture, IP based transport infrastructure
• Superior user experience
• Seamless connection to legacy networks (GSM, UMTS, 3GPP2/CDMA)
These improvements can be assessed with KPI measurements as defined in this docu-
ment.
The KPIs are defined for different service levels (see References 20. [3GPP 36.300],
and Figure 1):
a) Application Services
b) LTE E2E Network Service
c) Radio Bearer (RB) Service
d) IP based LTE Bearer Services: X2, S1, S5/S8 and External
The KPIs of group B (LTE E2E Network Service) characterize the quality of the LTE
network as IP transport service for applications. They are in the center of field perfor-
mance investigations, because they characterize the overall LTE network performance.
The Application Service KPIs of group A (Application Services) show the user-perceived
quality of application services as impacted by the LTE network. The KPIs of this group
are service specific. The following services are of common interest of mobile network
operators:
• Web Browsing
• Data Upload / Download
• Audio Video Streaming
• Conversational Audio Video (VoIP and Video Telephony)
This paper concentrates on the Data Upload / Download (File Transfer Protocol
(FTP)/Transmission Control Protocol (TCP)) application service, because it is the
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 8/70
8
DN0951948 Issue 01 DRAFT
LTE E2E Field Network Performance - Definitions of Key Performance Indicators
Id:0900d8058070a69b
Introduction
support of this service that stands in focus of the first commercial NSN LTE product
release.
In addition, KPIs for the radio link (RB Service, group C - Radio Bearer Service) are also
specified, because the radio link is the most critical component of the E2E transportpath. The other bearers, the E-UTRAN Radio Access Bearer (E-RAB) and Evolved
Packet System (EPS) bearers are composed of the RB Service and of the IP based
bearer services (of group D - IP based LTE Bearer Services: X2, S1, S5/S8 and Exter-
nal).
Monitoring the performance of all bearer services is necessary to fully understand the
E2E transport behavior of the LTE network and to optimize the end-user perceived
quality of application services running on top of it.
Figure 1 LTE Bearer Service Architecture
While the KPI definitions (see 3 Definitions of Key Performance Indicators (KPIs)) are
independent of LTE product releases, chapter 2.3 Measurement Scenarios focuses on
the first product release in order to restrict the number of possible scenario combinations(e.g. by not considering scenarios with Guaranteed Bit Rate (GBR) bearers).
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 9/70
DN0951948 Issue 01 DRAFT
9
LTE E2E Field Network Performance - Definitions ofKey Performance Indicators
Measurement Methods and Reference Conditions
Id:0900d8058072d090
2 Measurement Methods and Reference Condi-
tions
2.1 Introduction
The measurement method to be applied for field performance verification depends on
the specifics of the measured Key Performance Indicator (KPI). End-user perceived
KPIs are measured differently from network performance KPIs, and the conditions for
delay measurements are different from those for accessibility KPIs. For this reason, this
Section discusses measurement methods according to KPI categories.
Since the end-user experience of application services is mostly dependent on through-
put and latency, these aspects together with Handover (HO) scenarios constitute the
core part of field performance verification measurements.
While the scope of KPI definitions follows the performance goals of the LTE system (fordetails see References7. [3GPP25.913]), the selected methods and conditions of field
performance tests have to ensure that typical conditions in operational networks are
covered. Additionally, the measurement conditions also include relevant impairment
factors of the radio link, because the radio link has the most influence on E2E perfor-
mance. Especially, those characteristics of the Orthogonal Frequency-Division Multi-
plexing (OFDM) encoding technique, which have side effects for radio network
performance are considered in the selection of conditions, e.g.:
• Sensitivity to Doppler shift
• Sensitivity to frequency synchronization problems
• Sensitivity to multipath effects and cell area (urban, sub-urban, rural)
• Dependency on used bandwidth and frequency range
• Dependency on used multiple antenna techniques - Multiple Inputs Multiple Outputs
(MIMO) (transmit diversity, spatial multiplexing)
These aspects have a major bearing on the design of measurement scenarios. For
example, investigating the impact of Doppler shift on network performance, e.g. on user
and cell data rates, on delay and packet loss KPIs requires that high speed movement
becomes part of the measurement scenario. Another aspect would be the multipath
reception, which decreases the impact of the Doppler shift. Thus, the measurement
setup should also include space diversity if worst-case conditions are being discussed
measurements with these conditions have to show, how the investigated KPI figure
depends of the velocity and terminal category of the receiving user.
Some of the measurement scenarios might be much too costly to implement in live
network environments, or it may be impossible to ensure the desired radio conditions
during the whole duration of the test. In such cases, lab trials with simulated radio links
are acceptable as replacement of live network tests.
The combination of all different options for reference conditions and measurement sce-
narios would result in a very large number of test cases, which is simply not feasible. It
will be the task of test specifications to select options and scenarios out of the outlined
possibilities for a given LTE performance verification campaign considering the avail-
able resources and time constraints for test execution.
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 10/70
10
DN0951948 Issue 01 DRAFT
LTE E2E Field Network Performance - Definitions of Key Performance Indicators
Id:0900d805807292c5
2.2 Reference Conditions
Performance figures can only be interpreted correctly if the exact conditions under which
they have been acquired are known. This chapter describes the "reference conditions"
under which LTE field performance tests should be executed. They are formulated onan abstract level.
2.2.1 Distinguished Cell Positions
Operators are keen to know the distribution of throughput and latency KPI figures (e.g.
peak user data rate, cell UL/DL throughput, network delay, etc.) across the cells of a "ref-
erence cluster". However, it is impossible or would be associated with prohibitive high
costs to measure these KPIs in a live network environment in so many cell positions that
a Cumulative Distribution Function (CDF) can be calculated for 5%, 50%, or 95% of the
cell area. For this reason, it is proposed to replace the distribution function by KPI figures
measured on three characteristic positions of the cell (in the sequel, the term "cell" will
be used to denote both omni-directional and directional cells):
a) Very Good radio conditions (line of sight close to cell center within 50m, low path
loss)
b) Medium radio conditions (BTS distance of ~1/2 cell radius, channel with medium
noise conditions, medium path losses)
c) Cell Edge conditions (cell overlapping area, receive power from neighbor cells at
about the same level).
The radio conditions are formulated in terms of Signal to Interference and Noise Ratio
(SINR) and receive power levels. The measured KPI figures in Good, Medium and Cell
Edge positions replace their distribution across the LTE cell and will be considered as
their Peak, Average and Minimum values, respectively. The three variants are given bymean and 95% values calculated from all measurement samples, where the number of
samples is to be derived from the required confidence level.
If no suitable cell positions are available in the reference cluster, the tests can be
executed in the laboratory, where the radio conditions are to be simulated. If automated
test execution is possible, even the calculation of CDF function values at 5%, 50% and
95% distribution is an option.
This shows that often the full set of results can only be produced by combined field and
lab trials. In certain cases, even simulations might become necessary to accomplish
(incomplete) measurement results. For this reason, it is best practise to measure all E2E
field network KPIs in vendor lab before going into field tests with the operator.
The description of test conditions (see Table 2, row "Setup") specifies which test
methods are recommended.
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 11/70
DN0951948 Issue 01 DRAFT
11
LTE E2E Field Network Performance - Definitions ofKey Performance Indicators
Id:0900d805807292c5
2.2.2 Channel Models
Several factors of the radio link have influence on the E2E performance of application,
or LTE network services. Their importance is dependent on the category of the KPI to
be measured. The following parameters are checked for each scenario:
• UE category
• Antenna systems on UE and eNB
• Position and distribution of terminals in the cell, terminal movement
• Load conditions in the measured cell (UL/DL, data volume)
• Receive power level, interference and noise conditions, diversity and fading
• Cell area, or clutter type (urban, sub-urban, rural)
In order to reduce the number of possible combinations, mobile terminal speed and
radio propagation conditions for different environments and distances were combined in
"channel models". The relevant channel models for LTE are described in References 30.
Annex B of [3GPP36.521-1]. The following three channel models are defined: • EPA - Extended Pedestrian A
• EVA - Extended Vehicular A
• ETU - Extended Typical Urban
In addition, 3GPP proposes some combinations of channel models and Doppler fre-
quencies, which are complemented in Table 1 with scenarios for
• stationary UEs (EPA0, EVA0 and ETU0), and
• high speed train (ETU250)
The high speed train scenario with app. 250 km/h uses a carrier frequency of 900 MHz
(former GSM band) since it is assumed that initial LTE deployments will use this fre-
quency in rural areas. If the assumption is not true for a given customer trial, the Dopplerfrequency will change proportionally to the applied carrier frequency (500Hz @ 1.8 GHz,
583 Hz @ 2.1 GHz, 722 Hz @ 2.6 GHz). The current document uses only these models
as reference models.
Model Source Maximum Doppler Frequency
EPA0 0 Hz (stationary UE)
EVA0 0 Hz (stationary UE)
ETU0 0 Hz (stationary UE)
EPA5 3GPP36.521-1, Annex B; see 30. 5Hz
EVA5 3GPP36.521-1, Annex B; see 30. 5Hz
EVA70 3GPP36.521-1, Annex B; see 30. 70Hz
ETU70 3GPP36.521-1, Annex B; see 30. 70Hz
ETU250 250 Hz
(with 900 MHz carrier frequency)
ETU300 3GPP36.521-1, Annex B; see 30. 300Hz
Table 1 Channel Models and Doppler Frequencies
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 12/70
12
DN0951948 Issue 01 DRAFT
LTE E2E Field Network Performance - Definitions of Key Performance Indicators
Id:0900d805807292c5
Channel model conditions can be set in lab trials with simulated radio links precisely, but
they cannot be ensured in field network tests. In field trials, radio conditions with best
suit to the channel model are used to calibrate trial lab results.
2.2.3 Template for the Description of Reference Conditions
Table 2 includes the channel model together with other parameters to summarize the
essential conditions for measurement scenarios. The column "Reference Condition"
gives textual descriptions and possible settings as abstract values only.
The meaning of rows is as follows:
• "Antenna" identifies the number, type and modulation of antennas used on the UE
and on the eNB side. The reference UE configuration comprises two receive
antennas in DL and one antenna in UL. (However, it may not be freely selectable in
reference clusters of operational networks.)
• "The "Application" parameter identifies the test application as a real application, oran equivalent replacement (test script), which allows the collection of information
necessary to calculate / extrapolate the KPI figure.
• The "Setup" parameter identifies the environment, where the KPI can best be mea-
sured. It is not always the reference cluster of the live network. Some conditions, e.g.
SINR are easier to set in the lab.
– The "Reference Cluster" denotes a selected area of the operator´s live network
performance verification tests. In the cluster typical load conditions apply, as
generated by real users -stationary or mobile - and the radio conditions are
determined by cell location (rural, or city area, high, or low buildings, etc.). The
conditions are given and cannot be modified during the trial.
– The "Trial Lab" is either a vendor or operator lab used for executing performancetests, where exact measurement conditions can be ensured. The lab allows the
background traffic mix, load levels, and radio conditions to be precisely set. The
simulation of mobile users of high velocity is also possible.
– "Simulation" results are applied where field and even lab trials are expensive or
cannot be executed in the given time or budget.
• "Concurrent Load" describes the load which is to be generated in addition (i.e. back-
ground traffic) to the traffic produced by active test terminals in the measured cell.
Load conditions in neighbor cells are the operational load in field network clusters,
and are set to medium load in the lab.
• "Drive test "Routes" are defined for mobile terminals in live LTE environments. In
trial labs, terminal movement and hand-over are simulated (by varying the channel
conditions).
• "Backhaul Capacity" defines the necessary throughputs of transport links if their
values deviate from default and are important for test execution, e.g. to prevent a
transport link becoming bottleneck in user data rate, or cell throughput measure-
ments. It also defines other parameters, like Packet Delay (PD), Packet Delay Vari-
ation (PDV), Packet Loss Rate (PLR) and their expected values. These parameters
are observed in field trials, and are set to their maximal allowed values in the lab.
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 13/70
DN0951948 Issue 01 DRAFT
13
LTE E2E Field Network Performance - Definitions ofKey Performance Indicators
Id:0900d805807292c5
The number of test executions (sample size) is not defined as a parameter in the tem-
plate, but it is to be defined for each measurement scenario so that the calculated KPI
figure has the required statistical confidence.
Parameter Reference Condition
Terminal (UE)
Type Mobile or smart phone, laptop with LTE card, or w/
modem
Category UE categories 1-5 according to [3GPP36.306]; see 21.
Antenna Type: SISO, MIMO1*2, MIMO2*2, MIMO4*2, etc.
Modulation: QPSK, 16QAM, 64QAM
Position Good, medium and cell edge (for each terminal)
Distribution Equal, or non-equal distribution of "1 to n" UEs
Network Environment
Channel Model EPA, EVA, ETU with different Doppler frequencies
Backhaul Capacity Capacities of X2, S1-U, S1-MME transport l inks.
QoS Configuration Link weight and priority assigned to the flow(s)
Concurrent Load Unloaded, or concurrent UL/DL data volumes
Application Application service or traffic generation for LTE E2E,
EPS, E-UTRAN, EPC bearers.
Traces Trace points & data to be collected
Setup Trial Lab, Reference Cluster, or Simulation
Route In reference cluster, drive test route description.
Table 2 Description of Reference Conditions
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 14/70
14
DN0951948 Issue 01 DRAFT
LTE E2E Field Network Performance - Definitions of Key Performance Indicators
Id:0900d805807292ed
2.3 Measurement Scenarios
Measurement Scenarios describes the measurement methods and recommended sce-
narios according to KPI categories summarized in Table 3. The first column indicates
the respective QoS terminology defined by the ITU [ITU-T.E800], and referenced by3GPP in References 12. [3GPP32.410] and 17. [3GPP32.450] for LTE related KPIs.
The third and fourth columns indicate the selected terminology and explanation for its
usage in the current paper.
Since the KPI category is part of all KPI definitions (see KPI Definition Template ), it is
easy to lookup the corresponding measurement method to each KPI.
2.3.1 Throughput Measurements
Throughput is the primary metric for the characterization of the LTE radio technology
(Orthogonal Frequency Division Multiple Access (OFDMA) in DL, Single-Carrier Fre-
quency Division Multiple Access (SC-FDMA) in UL). It shows the data rate that can be
provided to users among different radio conditions. The throughput of an LTE cell is
expressed as a function of the number of concurrently active users and the amount of
aggregate traffic that the cell can still support. The throughput the LTE system can offer
depends on many factors:
• Channel environment (e.g. stationary or mobile, speed) and fading conditions.
• Reception conditions impaired by traffic load levels, and by interference between the
cells, in short by the user's SINR.
• Network layout, type of antenna.
• Position of users in the cell (implies e.g. path loss and fading).
ITU QoS
Category
3GPPKPI
Category
Terminology
Used
Meaningfor LTE Field Network
Performance
Accessibility Accessibility Accessibility Service accessibility, e.g. Attach, or
EPS Bearer Setup Request success
rates.
Reliabil ity Retainability Reliability Once established, the reliability of theservice, e.g. VoIP call, or EPS bearer
drop rates.
Integrity Throughput,
Latency
Throughput,
Latency
Promise to deliver the service with a
certain quality, level of conformance.
E.g. cell throughput, or user data rates,
packet delay.
Availability Availability not used Relationship between service up and
down times, e.g. cell, or LTE network
availability
not defined Mobility Mobility Different mobility related metrics, e.g.
hand-over success or drop rates, laten-
cies.
Table 3 KPI Categories
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 15/70
DN0951948 Issue 01 DRAFT
15
LTE E2E Field Network Performance - Definitions ofKey Performance Indicators
Id:0900d805807292ed
Thus, it is obvious that the location of the mobile terminal in the cell will strongly influ-
ences the data rate that can be experienced by the user. For this reason, the capacity
of an LTE cell is characterized by two distinguished metrics, one viewing capacity from
the operator's point of view (How many users can be supported by the cell?), the other
from the end-user perspective:
• Cell Throughput, and
• Peak User Data Rate
2.3.1.1 Cell Throughput
The metric "Cell Throughput" shows the sustainable aggregate cell capacity available to
a number of "n" users. In order to approximate a typical operating environment, the
users are distributed uniformly in the cell and use a typical mix of applications concur-
rently. The Figure 2 shows how users are arranged for cell throughput measurements.
Each user position is characterized by different combinations of path loss, receive
signal, and neighbor cell interference strengths.
Figure 2 Distribution of Users for Cell Throughput Measurements
The measurement is done with stationary terminals to avoid the impact of movement
and Hand-Over (HO). The latter implies that for those mobiles that are located at the cell
edge, HO must be suppressed on the UE or by the network. The proposed referenceconditions for cell throughput measurements are summarized in Table 4.
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 16/70
16
DN0951948 Issue 01 DRAFT
LTE E2E Field Network Performance - Definitions of Key Performance Indicators
Id:0900d805807292ed
The default backhaul capacity of S1-U transport links are to be controlled and modified
if necessary to prevent this link becoming a bottleneck during the measurement. The
modified value should become default for subsequent measurements.
Cell throughput measurements should be executed with an increasing number of UEs
starting with 1. While the first test case corresponds to the peak user data rate measure-
ment, the subsequent ones illustrate the dependency of this KPI on the number of active
Parameter Reference Condition
Terminal
UE Type Laptop with LTE card
UE Category Different categories, KPI specific
Antenna Types: SISO, MIMO 1*2, 2*2, …;
Position Good, medium, cell edge (see 2.2.1 Distinguished Cell Posi-
tions)
Distribution "1-n" terminals, equal distribution in the cell
Network Environment
Channel Model Stationary outdoor - EPA0, EVA0, ETU0 with different SINRs
Backhaul Capacity Capacities of X2, S1-U, S1-MME transport links.QoS Configuration Non-real-time QoS, equal weights and priorities
Concurrent Load Unloaded (no concurrent load)
Application Data upload / download (FTP); separate UL/DL and concurrent
transmission.
Traces UE, Uu, eNB, S1-U, … FTP Server
Setup Reference cluster in live network
Route N.a.
Table 4 Reference Conditions for Cell Throughput Measurements
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 17/70
DN0951948 Issue 01 DRAFT
17
LTE E2E Field Network Performance - Definitions ofKey Performance Indicators
Id:0900d805807292ed
users. The results (cell throughput and peak user data rate) are to be illustrated together
in a diagram like the example below:
2.3.1.2 Peak User Data Rate
The "Peak User Data Rate" metric describes the data capacity that is available to one
user in a cell. Data rate is the most important single factor which influences the end-user
experience of non-real-time application services, like web browsing, email, File Transfer
Protocol (FTP) up/downloads, and interactive gaming, due to its impact on (signaling
and data) transfer delays. The goal of field performance measurements is to investigate
the sensitivity of this metric to LTE impairment factors. The realistically achievable data
rates under concurrent load (i.e. with concurrently active users in the cell) in stationary
case are shown with cell throughput measurements as described in 2.3.1.1 Cell
Throughput.
Since the available peak user data rate is influenced by any or all of the following con-
ditions,
• Channel environment (mobility, stationary)
• Radio conditions (signal power, path loss, SINR)
• Aggregate cell load (i.e. intra-cell and inter-cell traffic)
• Restriction of user data rates (e.g. by terminal category)
• Link sharing weights (Quality of Service (QoS) configuration)
• Backhaul capacity
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 18/70
18
DN0951948 Issue 01 DRAFT
LTE E2E Field Network Performance - Definitions of Key Performance Indicators
Id:0900d805807292ed
the measurement scenarios include variations of these parameters. User data rates are
also measured in cell edge conditions, even including hand-over. The reference condi-
tions for user data rate tests are summarized in Table 5.
Ideally, the results of field or lab trial measurements of this KPI would be a full distribu-
tion of Peak User Data Rate values (a Cumulative Distribution Function (CDF)) across
the cell. For practical reasons, however, it is sufficient to produce the values in selected
cell positions as described in 2.2.1 Distinguished Cell Positions.
Radio positions (good, average and poor SINR) that are best suited to the channel
model of stationary users is selected. Mobility tests with up to 50 km/h can be executed
in field reference clusters, but higher speeds, e.g. high speed train with ~350 km/h are
to be executed in simulated a lab environment.
2.3.2 Latency Measurements
2.3.2.1 Round Trip Time Measurement
The benchmark measurement for finding the E2E latency of LTE access networks for
data applications is the Round Trip Time (RTT) measurement. RTT is measured with
the Ping application of the UE´s operating system or with a comparable measurement
tool. It records the time difference between sending an Internet Control Message
Protocol (ICMP) Echo Request to an IP host, and the reception of the corresponding
ICMP Echo Reply message.
The measurement can be executed between a mobile terminal and an IP host, which is
configured to respond to ICMP Echo Requests. This way, round trip delays can be
Parameter Reference Condition
Terminal
UE Type Smart phone and laptop with LTE card
UE Category Corresponding UE categories
Antenna Types: SISO, MIMO 1*2, 2*2, etc.
Position Good, medium, and cell edge (see 2.2.1 Distinguished Cell Positions)
Distribution One single terminal
Network Environment
Channel Model EPA, EVA stationary and mobile w/ 3, 50 km/h
Backhaul Capacity Capacities of X2, S1-U, S1-MME transport links.
QoS Configuration Non-real-time QoS, equal weights and priorities
Concurrent Load Unloaded
Application FTP upload, download; separate
Traces UE, eNB, FTP Server; Uu, S1-U, S1-MME, X1, ...
Setup Reference cluster in live network
Route Drive test route passing good and medium cell positions, and crossing cell borders.
Table 5 Reference Conditions for User Data Rate Measurements
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 19/70
DN0951948 Issue 01 DRAFT
19
LTE E2E Field Network Performance - Definitions ofKey Performance Indicators
Id:0900d805807292ed
measured between UE - eNB (LTE Uu), UE - S/P-GW (E-UTRAN Radio Access
Bearer(E-RAB), EPS) and UE - AS (LTE E2E), respectively. The measurement can
(and should) be executed in both directions.
The Ping application sends successive ICMP Echo Requests • either upon reception of the ICMP Echo Reply to the previous request,
• or after a predefined period of time (in the range of ~100ms to 1s)
This is done to ensure that only one Ping request/response is under way at any given
time. This way, the delay time will have no bias caused by queuing, or scheduling delays
along the transport path. The total number of measurements per cycle should be chosen
so that results have the required statistical confidence level. The recommended number
of measurements per IP host is ≥ 100.
If the measurement is executed over the radio link (LTE Uu), the first IP packet might
suffer additional delay caused by the dynamic allocation of radio resources to non-real-
time (non-GBR) bearers. For this reason, the round trip delay of the first packet should
not be considered in statistic calculations. On the other hand, the first ping measurement
can be used to determine UE state transition delays if the UE starts from non-registered
or idle states (from non-registered to registered, or from idle to connected states). The
periodicity of Pings needs to be selected so that the once allocated radio bearer is kept
over the whole duration of the measurement.
The test is to be done for stationary and mobile end users. The measurement with sta-
tionary UE should be performed under average radio link conditions with normal SINR.
For moving UEs, two different positions are to be selected, one in average radio link con-
ditions and normal SINR, and another in the cell overlapping area. It is important to
make sure that handovers between cells are avoided. The reference conditions should
be monitored during the test using UE performance measurements; also, relevant trace
data can be collected from the network.
In lab trials, the measurements need to be executed with and w/o concurrent load.
From the measurement results (samples) mean figures, standard deviation and 95%
delays need to be calculated.
2.3.2.2 Signaling and Media Delay Measurements
If the bandwidths of UL and DL bearers in RTT measurements are identical, the bench-
mark value of UL and DL latencies can be given as 1/2 RTT.
However, latency KPIs of signaling procedures on the network or application level (e.g.
Radio Access Bearer (RAB) service setup, or VoIP call setup flows) are often complexand include the exchange of more than one messages of different sizes. In addition, the
link capacities used in UL and DL are often different, e.g. in case of the application
services data download, gaming or video streaming.
In these cases, the RTT or 1/2 RTT figures are not sufficient to characterize the latencies
incurred by the mobile access network. Here, the time synchronization of network
elements is the central problem, since the start and stop triggers of the measurement lie
on different Network Elements (NE) and interfaces.
In case of measuring latencies over the radio link, the time needed to attach the mobile
to the network and to setup the necessary radio resources is a significant component.
In these cases, the measurement should be made with different initial states of the
mobile, e.g. power off, idle, connected and active states. LTE State Models lists the dif-
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 20/70
20
DN0951948 Issue 01 DRAFT
LTE E2E Field Network Performance - Definitions of Key Performance Indicators
Id:0900d805807292ed
ferent states of Mobile Terminations (MT) in 3GPP LTE specifications, see References
3. [3GPP23.401].
In field tests, latencies should be measured with the typical initial state of the mobile. The
initial state will depend on the measurement scenario, e.g. Idle for network attach, orConnected for EPS bearer setup.
Depending on the location of trigger points, end-user visible delays, LTE network
delays, different bearer (RB, E-RAB and EPS) delays, transport and NE processing
delays can be described similarly. The following aspects need to be specified for latency
measurements:
• Start and stop triggers of the media or signaling flow. The trigger point is defined by
the message type, NE and interface; it identifies where time stamps are to be
recorded.
• In case of measuring one-way delays, the method of time synchronization.
• Trace points and analysis tools. In addition to trace points in the network, traces may
be needed on mobile terminals when measuring end-user visible latencies, or E2E
LTE network delays.
2.3.2.3 Reference Conditions for Latency Measurements
Field measurements of latencies are requested under reference radio conditions (equal
to cell medium), which are specified by the acceptable ranges of UL/DL SINR and of
receive power levels. In addition, the following variations of conditions are planned (see
also Table 6):
• Unloaded versus loaded cell conditions
• Stationary UE versus mobile on the move
• Drive route for measurements with velocity from medium to cell edge position includ-
ing hand-over
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 21/70
DN0951948 Issue 01 DRAFT
21
LTE E2E Field Network Performance - Definitions ofKey Performance Indicators
Id:0900d805807292ed
Those samples that were taken in measurements during which the specified conditions
(reference, or explicitly modified ones) were not met, have to be excluded from the sta-
tistic calculations.
2.3.3 Service Accessibility Measurements
Accessibility KPIs characterize the accessibility of a service, or service element to its
users. They are expressed as a ratio of successful service requests to the total number
of attempts. (This group of KPIs corresponds to the category of "service accessibility
KPIs" in References 12. [3GPP32.410].) The KPI value is calculated as:
Success Rate = (# successful service requests / # total attempts) * 100 [%]
For this test to be meaningful, a significant number of samples need to be collected. E.g.
to get the success rate with 0.2% granularity, at least 500 samples are needed.
Service request attempts that are rejected due to insufficient access rights, bad user
authentication, or errors in request parameters are not considered and should be taken
out of the samples used for calculating the metric.
The measurements should be executed with the following variations of reference condi-
tions:
Parameter Reference Condition
Terminal
UE Type Mobile phone, smart phone, laptop with LTE card
UE Category UE category 1-5 according to [3GPP36.306]; see References 21.
Antenna One antenna: e.g. SISO
Position Medium, overlapping area
Distribution One terminal
Network Environment
Channel Model EVA0, EVA70 (stationary, mobile w/ ~50 km/h)
Backhaul Capacity Capacities of X2, S1-U transport links.
QoS Configuration Real-time and non-real-time QoS, the latter with equal weightsand priorities
Concurrent Load unloaded and loaded with concurrent UL/DL data of medium
volume (below cell target capacity)
Application Data upload download and real-time over EPS bearer.
Traces Trace points, send & receive packet counts
Setup Field Reference Cluster, Trial Lab
Route From radio position with reference condition (medium) to overlap-
ping area including hand-over.
Table 6 Reference Conditions for Latency Measurements
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 22/70
22
DN0951948 Issue 01 DRAFT
LTE E2E Field Network Performance - Definitions of Key Performance Indicators
Id:0900d805807292ed
• Application is selected according to the mobile network service procedure that is
tested. Test scripts are used for the automated execution of service requests, for the
evaluation of responses and for the calculation of the KPI figure.
• Mobile terminals in stationary and mobile use. • The mobile use includes drive routes among good and medium receive conditions,
as well as a route that crosses cell overlapping areas. In the latter case, hand-over
is executed. The frequency of handovers follows the assumptions of the NSN refer-
ence traffic model for LTE networks.
• Unloaded and loaded cell conditions. Concurrent load is to be generated in lab trials
with low and medium levels (as a certain percentage of the cell target load). In field
network tests, where a certain background load is given, load levels are to be traced
to allow for scaling (calibrating) of lab trial results.
The reference conditions are summarized in Table 7.
Due to the high number of test executions, the measurements can be executed in lab
environment with simulated air links. At least the stationary tests should be executed in
field network cluster. Automated test execution is needed in both environments.
In addition to the KPI figures calculated by the test script, test results should include
online performance measurement statistics, which are gained by monitoring the perfor-
mance counters of LTE network elements (i.e. on the eNB, MME, S-GW, P-GW). Please
Parameter Reference Condition
Terminal
UE Type Laptop with LTE card, or modem.
UE Category UE category according to [3GPP36.306]; see References 21.
Antenna One antenna: e.g. SISO
Position Good, medium and cell edge area.
Distribution n.a. (one terminal only)
Network Environment
Channel Model EVA0, EVA70: stationary, mobile w/ ~50 km/hBackhaul Capacity Capacities of X2, S1-U, S1-MME transport links.
QoS Configuration n.a.
Concurrent Load Unloaded or concurrent UL/DL data volumes of low and medium
levels.
Application Procedure dependent test script for automated execution of dif-
ferent applications.
Traces Data collected by test script. Monitoring of onl ine statistic coun-
ters.
Setup Field Network Cluster and Trial Lab.
Route Real drive test routes (in Cluster), or simulated with varying
channel models (in Lab).
Table 7 Reference Conditions for Success Rate Measurements
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 23/70
DN0951948 Issue 01 DRAFT
23
LTE E2E Field Network Performance - Definitions ofKey Performance Indicators
Id:0900d805807292ed
note, however, that the success ratio figures calculated from eNB counters will not be
identical to the success ratio measured by the test UE.
2.3.4 Reliability MeasurementsDue to similarities in their measurements, the following types of KPIs are discussed
together in this chapter.
• Service Drop Rates (Register, Attach, Service Request, etc.)
• Transport Error Rates (Block Error Rate (BLER), Packet Loss Ratio (PLR), Frame
Error Rate (FER), etc.)
The service drop rate KPIs are categorized as Retainability by 3GPP, and Reliability by
the ITU. The transport error rate KPIs are categorized as Integrity KPIs by 3GPP, and
by the ITU.
2.3.4.1 Transport Error Rates
These KPIs put the number of erroneous or lost data units to the overall number of data
units sent into relation. Depending on the payload of the transmitted data unit, the ratio
is given as BLER for radio blocks, FER for voice and video frames, and Packet Loss
Rate for IP packets. The chapter deals with the method of measuring PLR in the field.
Measurements of BLER and FER are planned for lab tests only; their methods will be
provided elsewhere.
E2E IP PLR is measured with the IP flow of an application service over a single EPS
bearer between the UE and the Application Server (AS) connected directly to the P-GW
(first hop IP router). The RAB is selected with appropriate QoS handling (i.e. nGBR
bearers for data applications and GBR bearers for real-time flows).
The measurement equipment for PLR is placed on the UE side and on the AS side to
measure PLR in both directions (UL and DL). It can either be a professional measure-
ment tool, or a simple test script.
PLR KPIs are measured across the following variations of conditions:
• Application service: data upload / download, and real-time (e.g. video streaming)
• Mobile terminal in stationary and mobile use
• The mobile use includes drive routes among good and medium reception conditions,
as well as a route that crosses cell overlapping areas. In the latter case, a handover
is executed.
• Operational load in source and destination cells of the field network cluster. Concur-
rent load generated in lab trials according to the reference traffic mix at low, medium
and high levels.
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 24/70
24
DN0951948 Issue 01 DRAFT
LTE E2E Field Network Performance - Definitions of Key Performance Indicators
Id:0900d805807292ed
One KPI value (measured sample) is calculated for each scenario as:
Packet Loss Rate = (# not received packets / # sent packets) * 100 [%]
The PLR is given as mean value over all samples of the same measurement scenario.
For this test to be meaningful, a significant number of samples needs to be collected.
For this reason, the test should run for at least one hour in the lab.
It is not necessary to make a detailed analysis of the reasons behind packet losses, as
it is difficult to store extremely large amounts of trace data. However, if professional test
equipment is available for this analysis, the measurement setup may include it.
Online performance statistics on PLR can be collected during the measurement if the
corresponding Performance Management (PM) counters are available on LTE network
elements (i.e. on the P-GW). Please note, however, that the PLR figures collected may
not be identical to the PLR measured on the test UE and on the AS.
Parameter Reference Condition
Terminal
UE Type Laptop with LTE card
UE Category UE category tbd. according to [3GPP36.306]; see 21.
Antenna One antenna: e.g. SISO
Position Good and medium in originating cell, drive across hand-over
area.
Distribution n.a. (one terminal only)
Network Environment
Channel Model EVA0, EVA70: stationary, mobile w/ ~50 km/h
Backhaul Capacity n.a.QoS Configuration n.a.
Concurrent Load Loaded & unloaded in source & destination cells
Application Data download (UDP) and video streaming.
Traces UE and eNB, mobility events
Setup Reference Cluster, Trial Lab.
Route Drive test route passing best and medium cell positions, and
crossing cell borders.
Table 8 Reference Conditions for Packet Loss Rate Measurements
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 25/70
DN0951948 Issue 01 DRAFT
25
LTE E2E Field Network Performance - Definitions ofKey Performance Indicators
Id:0900d805807292ed
2.3.4.2 Service Drop Rates
Attach, Service Request, Register, etc. are examples of service requests on different
service levels; service requests establish a relationship between the UE and the network
or application. The relationship is normally terminated with a "Release Request" by theUE. In case of network errors (e.g. handover failures), or insufficient radio resources the
relationship can be terminated unexpectedly. The ratio of such "abnormal" terminations
to the number of all established relationships is given by "drop rate" KPIs.
For certain services, like bearer creation or VoIP call, the service drop rate is also related
to the duration of the session. This is motivated by the fact that the possibility of network
errors, e.g. hand-over failures is higher if the service duration is longer. In such cases,
the service usage time should also be measured, and the drop ratio expressed in [1/s]
units instead of [%]. The current version of this document does not consider session time
for service drop rates. It will be added in the next update of this document.
In order to measure the ratio, the given service request has to be executed in a suffi-
ciently large number of times to assume statistical significance. E.g. each test scenario
should be executed ≥ 100 times. After the relationship has been established, common
operations of the service are to be executed for a typical period of service usage time.
The LTE traffic model can deliver input for this.
Due to the complexity of the test setup and the long time needed to execute the tests,
the measurements are better executed in a lab environment with simulated air links and
automated service execution.
The measurements are to be executed under the following varying conditions:
• Mobile terminal in stationary and mobile use
• The mobile use includes drive routes among good and medium reception conditions,
as well as routes that cross cell overlapping areas. In the latter case, a handover isexecuted. If executed in the trial lab, these conditions are set in simulated radio links,
i.e. the "drive route" is simulated.
• Unloaded and loaded cell conditions. Concurrent load is to be generated up to full
(or target) load level of the cell.
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 26/70
26
DN0951948 Issue 01 DRAFT
LTE E2E Field Network Performance - Definitions of Key Performance Indicators
Id:0900d805807292ed
2.3.5 Mobility (Handover) Measurements
Mobility KPIs from a group because all of them are related to hand-over procedures.
They could, however, be discussed in the Latency and Accessibility groups, as well,
because HO-related KPIs are either service interrupt time, or success ratio type KPIs.
A successful hand-over is understood to be any successfully performed make-before-
break procedure. For its measurement, a DL User Datagram Protocol (UDP) data
stream is generated as the only load during the whole measurement. The test is per-
formed in an overlapping area between two cells. The overlapping region of a cell is
defined by the level of radio conditions (DL SINR and receive power levels) which are
identical for neighboring cells. It can be expressed as the difference between receive
sensitivity levels of neighboring cells. I.e. at the border line of the overlapping region the
relative receive sensitivity level is 0 dB.
Drive routes need to be specified so that they begin in good radio conditions of the orig-
inating cell and proceed directly through the hand-off area and end in high signal
strength and good SINR in the other cell. Drive routes should proceed directly through
the hand-off area in order to prevent multiple handoffs caused by stop lights, stop signs,
traffic congestion as well as to prevent extending of the time period in the hand-over
region. Routes must be carefully selected to avoid areas where handoff is driven through
lack of coverage, which leads to breaks in connection. This is the most common cause
of so-called "false failures".
Parameter Reference Condition
Terminal
UE Type Laptop with LTE card
UE Category UE category tbd. according to [3GPP36.306]; see References
21.
Antenna One antenna: e.g. SISO
Position Good, medium and cell edge area.
Distribution n.a. (one terminal only)
Network Environment
Channel Model EVA0, EVA70: stationary, mobile w/ ~50 km/h
Backhaul Capacity Capacities of X2, S1-U, S1-MME transport links.QoS Configuration Link weight and priority assigned to the flow(s)
Concurrent Load Unloaded or concurrent UL/DL data volumes up to full cell load.
Application n.a.
Traces Trace points & data to be collected
Setup Trial Lab with simulated air link conditions.
Route The drive test routes are simulated using different channel
models and varying radio conditions (receive level, SINR).
Table 9 Reference Conditions for Service Drop Rate Measurements
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 27/70
DN0951948 Issue 01 DRAFT
27
LTE E2E Field Network Performance - Definitions ofKey Performance Indicators
Id:0900d805807292ed
In order to measure handover KPIs, the logging of mobility events is necessary in mobile
terminals. A successful handover is marked by an entry in the log, which carries the
identifier of the new cell (destination).
The HO KPIs are measured under the following variations of conditions (Table 10): • Unloaded versus loaded cell conditions for Intra RAT, ideal conditions in target cell
for Inter Radio Access Technology (RAT) (GSM, UMTS) hand-over scenarios.
• Mobile terminals with vehicular speed.
• Drive routes from good reception condition (in the originating cell) over hand-over
area to a good position on the destination cell.
The drive test has to be repeated at least 20 times to get the required level of statistical
confidence. All reasons for handover failures (no resources in target cell, service not
supported, etc.) should be excluded.
Parameter Reference Condition
Terminal
UE Type Smart phone, laptop with LTE card
UE Category UE category tbd. according to [3GPP36.306]; see References
21.
Antenna One antenna: e.g. SISO
Position Good/medium in originating cell directly across hand-over area to
good/medium position in new cell.
Distribution n.a.
Network Environment
Channel Model EVA0, EVA70: stationary, mobile w/ ~50 km/h
Backhaul Capacity n.a.
QoS Configuration n.a.
Concurrent Load Unloaded and loaded destination cell in UL/DL.
Application UDP data stream in DL with hand-offs acc. to reference traffic
model.
Traces MT, Source and Target eNBs, mobility events
Setup Reference Cluster, Trial Lab
Route In reference cluster, drive test route description.
Table 10 Reference Conditions for Handover Measurements
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 28/70
28
DN0951948 Issue 01 DRAFT
LTE E2E Field Network Performance - Definitions of Key Performance Indicators
Id:0900d805807295be
2.4 Measurement Setup
2.4.1 Field Network Cluster
For field performance verification tests of KPIs to be executed in live network environ-
ments, the following LTE Reference Cluster is recommended (see also Figure 3):
• Sufficient number and positioning of UEs as specified by the measurement scenario.
All UEs are inside of the cells of the reference cluster.
• eUTRAN including at least three eNBs, one MME, and an integrated S-GW/P-GW.
• Each eNB supporting three LTE cells, each with predefined minimum radio condi-
tions at cell borders.
• Provisioning of network equipment and setting of configuration parameters accord-
ing to the guidelines of the LTE network operator and agreed with the vendor.
The recommendation considers constraints of the NSN LTE product release.
The Reference Cluster environment must not change during the test campaign (no
changes to network configuration and parameter values).
Traces are to be activated on the UE, and on the network elements MME and S/P-GW
as required by the measurement scenario and reference conditions of the given KPI.
Trace data that can be collected on the interfaces LTE-Uu, S1, X1 and S11 are
described in References 14. [3GPP32.423]. Traces should be taken such that the inves-
tigated KPIs (e.g. latencies) are not impacted. If this cannot be avoided, the measure-
ments should be repeated without traces.
Figure 3 Reference Cluster for Field Performance Verification Tests
In addition to Figure 3, the final diagram of a test specification should contain indications
of:
• At least three eNBs with three cells each
• Trace points on all NE interfaces
• Monitoring station or test equipment with adequate protocol stack (e.g. K12xx) to
decode IP and radio network layer protocol messages • Test workstations (WS) to generate background load
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 29/70
DN0951948 Issue 01 DRAFT
29
LTE E2E Field Network Performance - Definitions ofKey Performance Indicators
Id:0900d805807295be
• WS for post-processing of traces
• Operations, Administrations and Maintenance Performance Management (OAM
PM) for monitoring counters, displaying of statistics
2.4.2 Trial Lab
A description of the Trial Lab should be given with the performance verification test
specifications.
2.4.3 Measurement Data Collection
The following ways of performance data collection are possible in KPI measurements:
• Collection of traces from terminals, network elements and network interfaces
• Collection and evaluation of counters from network elements by OAM PM
The preferred way of collecting performance data in a live network is via PM counters.It does not add additional load to the network and is a permanent source of information
that can be monitored all the times. However, PM counters do not exist for all categories
and for all individual KPIs. Typically, no counters are defined, or can be implemented for
latency KPIs. For this reason, terminal, server and network traces are also collected.
Since tracing puts additional processing load on these elements, traces are activated
only on operator demand and for the time of test execution. On the other hand, traces
include very detailed information on call level, which allows performance analyses of
individual mobiles and operations. With traces, it is possible to go further in monitoring
and optimization operations. For more details on LTE MMS and E-UTRAN traces, see
References 13. [3GPP32.421] and 14. [3GPP32.423].
2.4.3.1 Subscriber and Equipment Traces
Subscriber and UE Traces should be activated on the UE, and on the network elements
MME and S/P-GW of the Reference Cluster, as required by the scenario description
(see 2.2 Reference Conditions) of the given KPI measurement.
Collected measurement data is transferred to an external server for post-processing and
for evaluation. This data is necessary to check if the test run was executed w/o failure
(e.g. dropped call analysis) and if the measured figure can be considered for statistic cal-
culations and statements. Moreover, it helps to verify if the reference conditions were
met (e.g. RF coverage and capacity) during the whole measurement. Traces deliver
additional information for tuning and optimizing the Reference Cluster itself.
The capability to log data on any interface at call level for a specific user (e.g. Interna-
tional Mobile Subscriber Identity (IMSI)) or mobile type (e.g. International Mobile Equip-
ment Identity (IMEI)), or service initiated by a user makes it possible to get information
which cannot be deduced from Performance Measurements such as the end-user per-
ception of Quality of Service (QoS) during a call (e.g. requested QoS vs. provided QoS),
and correlation between protocol messages and RF measurements. Moreover, as
opposed to OAM Performance Measurements, which provide values aggregated on an
observation period (i.e. interval statistics), Subscriber and UE traces give instantaneous
values for a specific event (e.g. call, location update, etc.).
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 30/70
30
DN0951948 Issue 01 DRAFT
LTE E2E Field Network Performance - Definitions of Key Performance Indicators
Id:0900d805807295be
2.4.3.2 PM Counters
Performance measurements are mandatory in all field test scenarios. This provides per-
formance related information similar to operational environments.
The PM counters of special interest are given with the KPI description (see
3.1 Overview). This level of detail is provided in accordance with the availability of PM
counters in the latest LTE product release and in 3GPP specifications.
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 31/70
DN0951948 Issue 01 DRAFT
31
LTE E2E Field Network Performance - Definitions ofKey Performance Indicators
Id:0900d80580729612
2.5 Measurement Reporting
Some measurement scenario descriptions already contain hints about the form of the
expected measurement reports (e.g. in 2.3.1.2 Peak User Data Rate). A complete
description will be provided with the field verification test plan.
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 32/70
32
DN0951948 Issue 01 DRAFT
LTE E2E Field Network Performance - Definitions of Key Performance Indicators
Id:0900d8058072d166
Definitions of Key Performance Indicators (KPIs)
3 Definitions of Key Performance Indicators
(KPIs)
3.1 Overview
Key Performance Indicators (KPIs) are defined for application and for LTE network
services (see Figure 1):
a) Application Services
b) LTE E2E Network Service
c) Radio Bearer Services: E-RAB, RB
d) Transport Bearer Services: X2, S1, S5/S8 and External
e) LTE Network Elements
The Application KPIs are service specific. The following services are considered: • Web Browsing
• Data Upload / Download
• Audio Video Streaming
• Conversational Voice (VoIP)
The current paper focuses on the application service “Data Download/ Upload (FTP)
Service KPIs", and on “LTE E2E Network Service" KPIs in line with the feature scope of
the LTE release.
The performance indicators of the categories D and E are useful for the decomposition
of E2E performance figures. Their output can help to locate potential performance bot-
tlenecks in the field, and to optimize the overall performance of the LTE network. E.g.
the KPI values of category D can be the basis for Service Level Objectives (SLOs) for
IP backhaul networks if these are operated by third party ISPs. Because SLOs are very
important, maximum acceptable values are specified for them as assumptions for the
achievement of performance targets for application and LTE network service KPIs.
KPIs are distinguished according to the measurement setup, which should be used for
their verification. They are denoted "Lab" KPIs, if field network trials are not recom-
mended with them due to high costs or technical limitations. These KPIs are measured
in vendor or operator laboratories only. The other category of KPIs is called "Field" KPIs,
because they should be measured in Reference Network Clusters of operational net-
works. However, also "Field" KPIs have to be verified in the laboratory before measure-
ments are executed in the field. Lab results serve as benchmark figures and will be used
to calibrate (scale) field network measurement results.
After the overview of all KPI categories, detailed descriptions are given in
3.3 Application Services and 3.4 LTE Network Services.
3.1.1 Application Services
The full set of application service KPIs is summarized in Table 11. The following IP
based Packet Switched (PS) data services are characterized with a common set of
KPIs:
• Data Download / Upload (FTP)
• Web browsing (HTTP) • Email (POP/SMTP)
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 33/70
DN0951948 Issue 01 DRAFT
33
LTE E2E Field Network Performance - Definitions ofKey Performance Indicators
Definitions of Key Performance Indicators (KPIs)
Id:0900d8058072d166
The Data Download / Upload service is selected for detailed investigations and verifica-
tion in the LTE network, because of the Transmission Control Protocol (TCP) windowing
mechanism.
3.1.2 LTE Network Services
3.1.2.1 LTE E2E Network Service
The full set of LTE E2E network service KPIs is summarized in Table 12. Some of the
KPIs are defined as "Lab", which means that they are verified in in-house (or vendor)
labs, but not in the field.
# KPI Name KPI
Category
Trial
Network
Chapter
PS Data Services (FTP, HTTP, etc.)
1 Service Accessibility Ratio [%] Accessibility Field 3.3.1.1 (FTP) Service Accessibility Ratio
2 Completed Session Ratio [%] Reliability Field 3.3.1.2 (FTP) Completed Session Ratio
3 Service Access Time [s] Latency Field 3.3.1.3 (FTP) Service Access Time
4 User Data Rate [Mbps] Throughput Field 3.3.1.4 (FTP) User Data Rate
Table 11 Overview of Application Service KPIs
# KPI Name KPI
Category
Trial
Network
Chapter
LTE E2E Network Service, Control
Plane
5 Attach Time [ms] Latency Field 3.4.1.1 Attach Time
6 Attach Success Rate [%] Accessibility Field 3.4.1.2 Attach Success Rate
7 Service Request (EPS) Time [ms] Latency Field 3.4.1.3 Service Request Time (EPS),
UE Initiated,
3.4.1.4 Service Request (EPS) Time,
Network Initiated
8 Service Request (EPS) Success Rate
[%]
Accessibility Field 3.4.1.5 Service Request (EPS)
Success Rate9 Service (EPS) Drop Rate [%] Reliability Field 3.4.1.6 Service (EPS Bearer) Drop
Rate
10 Handover Procedure Time [ms] Mobility Lab 3.4.1.7 Handover Procedure Time
11 Handover Success Rate [%] Mobility Field 3.4.1.8 Handover Success Rate
12 Paging Time [ms] Latency Lab 3.4.1.9 Paging Time
13 Paging Failure Rate [%] Accessibility Lab 3.4.1.10 Paging Failure Rate
LTE E2E Network Service, User Plane
Table 12 Overview of LTE E2E Network Service KPIs
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 34/70
34
DN0951948 Issue 01 DRAFT
LTE E2E Field Network Performance - Definitions of Key Performance Indicators
Id:0900d8058072d166
Definitions of Key Performance Indicators (KPIs)
3.1.2.2 Radio Bearer Services
14 (LTE) Round Trip Time (RTT) [ms] Latency Field 3.4.1.11 (LTE) Round Trip Time
15 (LTE) User Data Rate [Mbps] Throughput Field 3.4.1.12 (LTE) User Data Rate
16 (LTE) Packet Loss Rate (PLR) [%] Reliability Field 3.4.1.13 (LTE) Packet Loss Rate
# KPI Name KPI
Category
Trial
Network
Chapter
Table 12 Overview of LTE E2E Network Service KPIs (Cont.)
# KPI Name KPI
Category
Trial
Network
Chapter
Radio Bearer Service, User Plane
17 (RB) Packet Loss Rate UL / DL [%] Reliability Lab 3.4.2.1 (RB) Packet Loss Rate
18 (RB) User Data Rate [Mbps] Throughput Field 3.4.2.2 (RB) User Data Rate
19 Cell Throughput [Mbps] Throughput Field 3.4.2.3 Cell Throughput
20 (RB) Service Interrupt Time (HO) [ms] Mobility Lab 3.4.2.4 (RB) Service Interrupt Time
(HO)
Table 13 Overview of LTE Radio Bearer Service KPIs
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 35/70
DN0951948 Issue 01 DRAFT
35
LTE E2E Field Network Performance - Definitions ofKey Performance Indicators
Id:0900d80580729749
3.2 KPI Definition Template
For each KPI, relevant information is provided according to the structure shown in Table
14. The fields in the KPI definition are marked as mandatory, or optional according to
References 12. [3GPP32.410].
The "KPI Object" field identifies the service, or network element to be characterized:
• Services: AS or one of the 3GPP bearer services: E2E network (LTE), EPS, E-RAB,
RB, X2, S1, S5/S8 and External. The bearer services are uniquely defined by their
end-points, i.e. by the network interfaces of the LTE architecture.
• Network Elements: eNB, MME, S/P-GW.
The following "KPI Categories" are used: Accessibility, Reliability (Retainability),Latency, Throughput, Mobility and Quality.
The PM measurements and KPIs of special interest are given with indication of their
relationship to the field network KPI, e.g. if and how to calculate the field network KPI
value out of them.
KPI Name M Mandatory long name of the KPI.
KPI Type, Category, Object, Unit M Type: Ratio, Mean, CDF, etc. [3GPP32.814]; see References 19.
Category: Accessibility, Reliability, Latency, Mobility, etc.
Object: Application Service, E2E, E-RAB, RB, X1, etc.
Unit: %, ms, Mbps, Erlang, etc.
Definition M Description of the KPI. End-user or network view. Basic information
about begin and end trigger points.
Methodology of measuring O Test environment: field or lab test, or simulation.Measurement method:
according to KPI Category, plus specifics for the KPI (if any).
Assumptions and pre-conditions O Important information for measurement execution, i.e. scenario
description including terminal category, channel model, load condi-
tions, etc.
Formula (Logical) M Provided for KPIs, e.g. Success Rate & Drop Rate which are calculated
from some other metrics (measured)
Message flow, trigger points O Provided for latency KPIs, for monitoring of procedure correctness, and
for success rate KPIs to indicate trigger points where PM counters are
incremented.
Related KPIs O Reference to related KPIs
Related PM counters O Reference to related domain specific PM counters with indication of
how to aggregate them to the KPI.
Table 14 KPI Definition Template
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 36/70
36
DN0951948 Issue 01 DRAFT
LTE E2E Field Network Performance - Definitions of Key Performance Indicators
Id:0900d805807297e9
3.3 Application Services
3.3.1 Data Download / Upload (FTP) Service KPIs
Download of reference files with different sizes from the Internet, e.g. SW images, pic-
tures, or video, as well as upload of the same files by FTP. For FTP protocol details, see
References 33. [RFC979].
3.3.1.1 (FTP) Service Accessibility Ratio
(FTP) Service Accessibility Ratio
KPI Category, Object, Type, Unit M Category: Accessibility, Object: AS (UE, FTP server),
Type: Ratio, Unit: [%]
Definition M The service accessibility ratio denotes the probability that the user canestablish the necessary bearer (EPS) and access the FTP service suc-
cessfully (see References 39. [ETSI102.250-2]).
Service access covers starting the FTP client on the UE, setting up
mobile access and creating a TCP connection to the FTP server.
Measurement method O Accessibility measurement, see 2.3.3 Service Accessibility Measure-
ments. Field and lab trial.
Assumptions, pre-conditions O Conditions for accessibility measurements see 2.3.3 Service Accessi-
bility Measurements
Formula (logical) M
Related KPIs O (FTP) Completed Session Ratio
Related PM counters O No PM counters exist for this KPI
Message flow, trigger points O Trigger points:
• Start: ftp get / put command issued on the UE
• Stop: first data byte sent / received by the UE, or ftp indicates
network error (e.g. timeout expired)
FtpCmdSR
number_of(successful_ftp_commands)
number_of(total_ftp_commands)--------------------------------------------------------------------------------------------------------- 100%⋅=
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 37/70
DN0951948 Issue 01 DRAFT
37
LTE E2E Field Network Performance - Definitions ofKey Performance Indicators
Id:0900d805807297e9
3.3.1.2 (FTP) Completed Session Ratio
3.3.1.3 (FTP) Service Access Time
Figure 4 shows the phases of a data download / upload (FTP) service session using
GPRS access network as example (see References 39. [ETSI102.250-2]). In case of an
LTE network, GPRS would be replaced by LTE and an EPS bearer would correspond
to the "PDP-Context". The full data download / upload session is subdivided into the
phases Service Access and Data Transfer.
In this document, the Service Access Time is defined as independent KPI. It corre-
sponds to the phase "IP Service Access" on the Figure 4. (The EPS bearer setup time
is defined as LTE E2E network KPI in 3.4.1.3 Service Request Time (EPS), UE Initi-
ated.) The trigger points for the Service Access Time KPI are defined on two levels:
• end-user interface (UI), i.e. on command line level, and
• TE - MT interface, i.e. on UE R interface level.
On the command line level, the end of the mobile network access phase is recognized
by the ftp> prompt. By this time, the FTP client is initialized, the terminal is attached to
the LTE network and the EPS bearer has been created. The Session is started by
issuing the "ftp get / put" command and terminates with the display of the results of the
data transfer (e.g. the # of bytes transmitted.). After the "get / put" command has been
(FTP) Completed Session Ratio
KPI Category, Object, Type, Unit M Category: Availability, Object: AS (UE, FTP server),
Type: Ratio, Unit: [%]
Definition M The completed session ratio is the proportion of completed FTP
sessions and sessions that were started successfully (see References
39. [ETSI102.250-2]). The session is not completed if a predefined
timer expires, or if a network failure has occurred that could not be
repaired by automatic restarts.
Measurement method O Availability measurement, see 2.3.3 Service Accessibility Measure-
ments. Field and lab trial.
Assumptions, pre-conditions O Conditions for availability measurements see 2.3.3 Service Accessibil-
ity Measurements.The size of the file used for data upload / download, or the session
duration time is to be specified for a concrete requirement on the target
value of this KPI.
Formula (logical) M
Related KPIs O (FTP) Service Accessibility Ratio
Related PM counters O No PM counters exist for this KPI.
Message flow, trigger points O Trigger points:
• Start: ftp get / put command issued on the UE.
• Stop: after error free execution of the command, ftp returns with a
prompt and indicates the number of bytes transmitted.
FtpSessionSRnumber_of(completed_sessions)
number_of(successfully_started_sessions)------------------------------------------------------------------------------------------------------------------- 100%⋅=
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 38/70
38
DN0951948 Issue 01 DRAFT
LTE E2E Field Network Performance - Definitions of Key Performance Indicators
Id:0900d805807297e9
be issued, a TCP connection to the FTP server is created and the data transfer phase
begins.
Trigger events of the Service Access Time KPI cannot be seen on the terminal UI. In
order to find them, it is necessary to analyze the message flows on the R interface level.The Session (and the IP Service Access phase, see Figure 4) begins with setting up the
TCP connection to the FTP server and terminates with the transmission (send, or
receive) of the last IP packet with file content payload. The phases Service Access and
IP Service Access end, and the Data Transfer phase begins with the transmission (send,
or receive) of the first IP packet with file content payload.
Figure 4 KPIs for Packet Switched Data Services
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 39/70
DN0951948 Issue 01 DRAFT
39
LTE E2E Field Network Performance - Definitions ofKey Performance Indicators
Id:0900d805807297e9
3.3.1.4 (FTP) User Data Rate
(FTP) Service Access Time
KPI Category, Object, Type, Unit M Category: Latency, Object: AS (UE, FTP server),
Type: Mean and max (95%) values, Unit: [s]
Definition M It is the time period needed to access the FTP service successfully,
from starting the ftp client to the point of time when the first data packet
is sent or received (see References 39. [ETSI102.250-2]). It is
assumed that the UE is already attached to the mobile network and an
EPS bearer exists.
Measurement method O Latency measurement, see 2.3.2 Latency Measurements.
Assumptions, pre-conditions O Conditions for latency measurements see 2.3.2.3 Reference Condi-
tions for Latency Measurements.
Formula (logical) M
Related KPIs O (FTP) Session Time, Service Request (EPS) Time
Related PM counters O No PM counters exist for this KPI.
Message flow, trigger points O Trigger points DL/UL:
• Start: "ftp get / put" command issued on the UI
• Stop: first data packet containing file content received / sent on the
R interface.
FtpServiceAccessTime [s] tcontent sent or received tftp command started–=
(FTP) User Data Rate UL / DL
KPI Category, Object, Type,
Unit
M Category: Throughput, Object: AS (UE, FTP server),
Type: Min (CDF 5%), Mean (CDF 50%), Max (CDF 95%) values, Unit: [Mbps]
Definition M After the connection to the FTP server has been successfully established, the
parameter describes the average data transfer rate measured over the data
transfer phase (see Figure 4). This means that successful Service Access
(availability of an EPS bearer and connection to the FTP service) is a prereq-
uisite to the data transfer. The data transfer phase should also conclude suc-
cessfully.
The data rate is measured on application level in UL/DL direction.
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 40/70
40
DN0951948 Issue 01 DRAFT
LTE E2E Field Network Performance - Definitions of Key Performance Indicators
Id:0900d805807297e9
It is to be noted that the measurement of the User Data Rate (FTP) KPI (named after
[ETSI102.250-2]; see References 39.) is executed as a single user measurement
without concurrent load in the cell, since the goal is to find the maximum available FTP
data rate for one user.
Measurement method O Throughput measurement, see 2.3.1.2 Peak User Data Rate. Single user
measurement both in field and lab trials.
In order to eliminate the impact of service startup times, the measurement
begins with the transmission of first IP packets with file content payload.Since the data rates can vary strongly from session to session, a sufficiently
large number of measurement executions is needed (>10) to get statistical
confidence.
The size of the large reference file should be selected such that the data
transfer phase does not take less then thirty seconds. Optionally, a small ref-
erence file can be measured in addition, but the number of executions should
be the same.
The measured User Data Rates should be displayed in diagrams as a
function of time to show the impact of FTP slow start.
Assumptions, pre-conditions O Conditions for capacity measurements see 2.3.1.2 Peak User Data Rate.
The FTP server will be connected directly to the SGi interface. Additionaldelays or bottlenecks between the S/P-GW and the server should be
excluded.
The initial status of UE before starting the ftp application:
EMM-REGISTERED, ECM-CONNECTED.
The FTP application will be executed in binary mode, and the file to be trans-
ferred will also be binary.
Formula (logical) M
Related KPIs O (FTP) Service Access Time, (FTP) Session Time
Related PM counters O No PM counters exist for this KPI.
Message flow, trigger points O The average throughput is measured from the opening of the data connec-
tion to the end of the successful transfer of the content (file, e-mail or web
page).
The trigger points for the Data Transfer phase can be seen in Figure 4. For
more details see References 39. [ETSI102.250-2], chapters 4.6.1.7 and
4.6.2.7.
FtpMeanDataRateUL/DLtrans ferred_data_volume_UL/DL[bytes] 8⋅
transfer_time [s]-----------------------------------------------------------------------------------------------------------------------=
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 41/70
DN0951948 Issue 01 DRAFT
41
LTE E2E Field Network Performance - Definitions ofKey Performance Indicators
Id:0900d8058072981f
3.4 LTE Network Services
This chapter includes detailed KPI definitions for LTE Network Services.
3.4.1 LTE E2E Network KPIs
3.4.1.1 Attach Time
Attach Time, Control Plane
KPI Category, Object, Type, Unit M Category: Latency, Object: End-user, LTE (UE R interface),
Type: Mean and max (95%) value, Unit: [s]
Definition M With Attach, the mobile terminal registers at the LTE network. At the
end of the procedure the UE is authenticated, and a default (nGBR)
bearer is established.The Attach Time is the interval between the connection request and the
reception of a positive response by the UE.
Measurement method O Latency measurement, see 2.3.2 Latency Measurements. Field and lab
trial. Stationary users only.
Assumptions, pre-conditions O Conditions for latency measurements, see 2.3.2.3 Reference Condi-
tions for Latency Measurements. UE status before measurement:
EMM-DEREGISTEREDMAP inter-working according to [3GPP29.002],
see References 10., is assumed between MME and HSS (no Diame-
ter).
Formula (logical) M tbd.: mean value and 95% from all measured samples
Message flow, trigger points O Trigger points on UE R interface (see Figure 5, References 3.
[3GPP23.401]):
• Begin: 1. RRC CONNECTION REQUEST sent by UE (see also
References 26. [3GPP36.331]) carrying L3 NAS: Attach Request in
its body.
• End: 21. RRC DIRECT TRANSFER message with L3 NAS: Attach
Complete in its body sent by UE (see References 4.
[3GPP24.008]).
Related KPIs O a) Attach Success Rate.
b) No 3GPP defined KPI for Attach Time found (exists).
c) Attach Setup Time KPI in [ETSI102.250-2], see References39.
chapter 4.6.3.2.
Related PM counters O Attach Time related measurements are neither defined in 3GPP
Release 8 nor in LTE reference documents. However, the same trigger
events for counting Attach requests and successful attachment com-
pletions can be used as Begin and End triggers for the measurement of
elapsed time on the eNB.
Attach Time [s] t At ta ch Complete t At ta ch Request–=
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 42/70
42
DN0951948 Issue 01 DRAFT
LTE E2E Field Network Performance - Definitions of Key Performance Indicators
Id:0900d8058072981f
Figure 5 Attach Procedure w/ Initial EPS Bearer Establishment
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 43/70
DN0951948 Issue 01 DRAFT
43
LTE E2E Field Network Performance - Definitions ofKey Performance Indicators
Id:0900d8058072981f
3.4.1.2 Attach Success Rate
Attach Success Rate, Control Planee
KPI Cate-
gory, Object,
Type, Unit
M Category: Accessibility,
Object: LTE (UE R, eNB Uu interface),
Type: Ratio, Unit: [%]
Definition M The Attach Success Rate is defined as the ratio between the number of successful registra-
tions and the number of all requests. This is the probability that a user can attach to the LTE
network at any moment of time.
The calculated success ratio figure excludes attach requests, which are rejected by authen-
tication failures. On the other hand, network attach requests which are terminated by timer
expiry (due to the unaccessibility of some LTE resource) are considered as unsuccessful reg-
istrations.
If the success rate is calculated on the eNB by counting incoming RRC requests, RRC CON-
NECTION REQUEST retries are to be excluded, since they would increase the overallnumber of establishment attempts, and thus reduce the success ratio.
Measurement
method
O Accessibility measurement, see 2.3.3 Service Accessibility Measurements. Field and lab
trial. Stationary users only.
For defining the success ratio, the same series of "Attach Time" measurements can be used.
The KPI cannot be measured as end-user perceived ratio on certain mobiles, which do not
indicate network attachment status to the end-user.
Assumptions,
pre-conditions
O Conditions for accessibility measurements see 2.3.3 Service Accessibility Measurements.
See also conditions in 3.4.1.1 Attach Time.
Formula
(logical)
M
Message flow,
trigger points
O Trigger events on UE R / eNB Uu interface (see Figure 5):
• Success: 21. RRC DIRECT TRANSFER message with L3 NAS: Attach Complete in its
body (see References 4. [3GPP24.008]) sent by the UE / received by eNB
• Attempts: 1. RRC CONNECTION REQUEST messages (with cause Attach Request)
sent by UE / received by eNB.
Related KPIs O a) Attach Time, Service Request (ESP) Success Rate.
b) E-RAB Accessibility" KPI in [3GPP32.450], see References17. chapter 6.1.1. Since an
SRB and S1 connection, as well as initial EPS bearer(s) have to be created during the
Attach procedure (see Figure 5), its success rate is dependent of the success rates ofthose component bearers. The success rate of initial EPS bearers is given by the follow-
ing expression (for “additional” EPS bearers, refer to the 3.4.1.5 Service Request (EPS)
Success Rate):
c) See "Attach Failure Ratio" KPI in References39. [ETSI102.250-2], Chapter 4.6.3.1
defined as the inverse probability of successful network attachment.
LTENwAttSRnumber_of_successful_attachments
number_of_all_attempts-------------------------------------------------------------------------------------------------- 100%⋅=
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 44/70
44
DN0951948 Issue 01 DRAFT
LTE E2E Field Network Performance - Definitions of Key Performance Indicators
Id:0900d8058072981f
Related PM
counters
O a) Related 3GPP measurements are defined in [3GPP32.425],see References15., for eNB:
Signaling Radio Bearer (SRB) creation:
• Attempts: RRC.ConnEstabAtt.Cause = Attach
• Success: RRC.ConnEstabSucc.Cause = Attach • Failures: RRC.ConnEstabFail.Cause = Attach
UE associated logical S1 connection (S1-AP):
• Attempts: S1SIG.ConnEstabAtt
• Success: S1SIG.ConnEstabSucc
Initial EPS setups during Attach:
• Attempts: SAEB.EstabInitAttNbr.QCI
• Success: SAEB.EstabInitSuccNbr.QCI
where QCI identifies the SAE Bearer level quality of service class.
• Failures: SAEB.EstabInitFailNbr.Cause
where Cause identifies the cause resulting in the setup failure.
b) The corresponding eNB counters are:
• Attempts: SIG_CON_EST_ATT due to Attach
• Successful attachments: SIG_CON_EST_SUCC
• Failures: SIG_CON_EST_FAIL
For the calculation of success, or failure ratios, it is necessary to have identical Causes
for _ATT, _SUCC and _FAIL types of counter groups.
Since the EPS_SETUP_ATT/_SUCC counters do not differentiate between initial and
subsequent EPS bearer setups, they cannot be set in direct relation to the Attach SR KPI.
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 45/70
DN0951948 Issue 01 DRAFT
45
LTE E2E Field Network Performance - Definitions ofKey Performance Indicators
Id:0900d8058072981f
3.4.1.3 Service Request Time (EPS), UE Initiated
Service Request Time (EPS Bearer Setup), UE Initiated, Control Plane
KPI Category, Object, Type, Unit M Category: Latency, Object: EPS (UE R, eNB Uu interface),
Type: Mean and max (95%) value, Unit: [s]
Definition M It is the time taken by the LTE network to setup an EPS bearer on
request by the UE. The EPS bearer has to be created before IP packets
can be exchanged, i.e. the EPS bearer is a transport link over the
mobile access (LTE) network.
The setup times of E-RAB and RB bearers are part of the EPS setup
time and can be defined using the same sequence of measurements.
The UE Initiated EPS Bearer Setup Time is the interval between the
submission of the message RRC CONNECTION REQUEST (see Ref-
erences 26. [3GPP36.331]) and the reception of the RRC CONNEC-
TION RECONFIGURATION response on the UE. The Bearer Setupprocedure is based on the Service Request procedure, see References
3. [3GPP23.401] and 20. [3GPP36.300].
Measurement method O Latency measurement, see 2.3.2 Latency Measurements. Field and lab
trial. Stationary users only. Dedicated, or default EPS bearer setup.
The response time can be measured on the UE R interface, or on the
eNB Uu interface. The latter metric shows the delay component of the
eUTRA network only.
End-user delays cannot be measured, since terminals do not indicate
the corresponding state change toward the end-user.
Assumptions, pre-conditions O Conditions for latency measurements see 2.3.2.3 Reference Condi-
tions for Latency Measurements.
UE status before measurement: EMM-REGISTERED , but otherwise
different initial states (idle, dormant or active).
Formula (logical) M Mean value and 95% from all measured samples.Service Request
Message flow, trigger points O The message flow diagram is shown in Figure 6 and Figure 7. The user
initiated EPS bearer setup time is measured between the trigger points
BUE and EUE on the UE R interface:
• Begin: RRC CONNECTION REQUEST sent by UE• End: corresponding RRC CONNECTION RECONFIGURATION
COMPLETE message or first PDU in UL sent by the UE (see Ref-
erences 4. [3GPP24.008]).
Related KPIs O a) Attach Time, network initiated Service Request (EPS Bearer Setup)
Time.
b) No 3GPP defined KPI for EPS bearer setup found (exists).
Service Request Time [s] tRRC_Reconfig tRRC_Request–=
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 46/70
46
DN0951948 Issue 01 DRAFT
LTE E2E Field Network Performance - Definitions of Key Performance Indicators
Id:0900d8058072981f
Related PM counters O The eNB delay budget of the overall Service Request Time can be
measured using two counters of [3GPP32.425], see References 15.
• Mean EPS Setup Time: SAEB.EstabTimeMean.QCI
• Max EPS Setup Time: SAEB.EstabTimeMax.QCI
The latency is measured between the trigger points T1, T2 as can be
seen in Figure 7.
Service Request (EPS) Time related measurements are not defined in
LTE product documents. However, the trigger events
EPS_SETUP_ATT and EPS_SETUP_SUCC could be used as start
and end triggers for elapsed time measurements on the eNB.
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 47/70
DN0951948 Issue 01 DRAFT
47
LTE E2E Field Network Performance - Definitions ofKey Performance Indicators
Id:0900d8058072981f
Figure 6 EPS Bearer Setup Procedure, Part 1
D
LTE-Uu S1- MME S11 S6a
UE eNodeB MME S-GW / P-GW HSS
S1-AP: UL NASTRANSPORT
(L3: AUTHENTICATION AND CIPHERING
REQUEST)
Begin measurementRRC CONNECTION
REQUEST
MAP: Send Authentication Info
Setup SRB1
RRC CONNECTION
SETUP
S1-AP: INITIAL UEMESSAGE
(L3 NAS: SERVICEREQUEST)
MAP: Send Authentication Info Ack
RRC: DL INFORMATIONTRANSFER
(L3: AUTHENTICATION AND CIPHERING
REQUEST)
If authentication
vectors are not
available in MME,
authenticate UE
RRC: UL INFORMATIONTRANSFER
(L3: AUTHENTICATION AND CIPHERING
RESPONSE)
S1-AP: DL NASTRANSPORT
(L3: AUTHENTICATION AND CIPHERING
RESPONSE)
If authentication
timer has expired,
setup security
association for user
plane RB
UE EMM-REGISTEREDand ECM-IDLE
RRC CONNECTION
SETUP COMPLETE
(L3 NAS: SERVICEREQUEST)
L2 SCTP: Create BearerRequest
L2 SCTP: Create BearerResponse
Activate EPS
bearer setup in
ePC
DL data
L2 SCTP: DL DATANOTIFICATION
BN
BUE
Paging Procedure
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 48/70
48
DN0951948 Issue 01 DRAFT
LTE E2E Field Network Performance - Definitions of Key Performance Indicators
Id:0900d8058072981f
Figure 7 EPS Bearer Setup Procedure, Part 2
The service request times for UE and for network initiated EPS bearer setups are
measured between the trigger points BUE and EUE, and BN and EN, respectively.These trigger points are colored green in the diagram.
The trigger points T1 and T2 denote events, which increment the PM counters
EPS_BEARER_ATT and EPS_BEARER_SUCC on the eNB. They are colored yellow.
These counters can be used on the PM server to calculate EPS bearer setup success
rates as seen by the eNB. Please note that this value will be different from that of the
Service Request Success Rate KPI defined in this document (see 3.4.1.5 Service
Request (EPS) Success Rate). The eNB counters calculate the RB (and not the EPS)
setup rate.
UE eNodeB MME S-GW / P-GW HSS
LTE-Uu S1- MME S11 S6a
S1-AP: INITIAL
CONTEXT
SETUP REQUEST
PM Counter:
EPS_SETUP_ATT
RRC SECURITY MODE
COMMAND
RRC CONNECTION
RECONFIGURATION
COMPLETE
S1-AP: INITIAL
CONTEXT
SETUP COMPLETEL2 SCTP: Update Bearer
Request
L2 SCTP: Update Bearer
Response
First UL PDU
T1
T2
RRC CONNECTION
RECONFIGURATION
RRC SECURITY MODE
COMMAND COMPLETE
Setup user
plane RB
Means “Delay DL
Packet Notification
Request” to stop/
enable DL data
transfer PM Counter:
EPS_SETUP_SUCC
End measurement
First DL PDU
EUE
EN
S1-AP: INITIAL
CONTEXT
SETUP FAILURE
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 49/70
DN0951948 Issue 01 DRAFT
49
LTE E2E Field Network Performance - Definitions ofKey Performance Indicators
Id:0900d8058072981f
3.4.1.4 Service Request (EPS) Time, Network Initiated
The network initiated Service Request Time is not described from the UE point of view,
because the delay component seen on the UE is not the complete time the downlink
data packet is delayed before it can be forwarded from the P-GW to the UE. The network
initiated EPS bearer setup time is dependent on the mobility state of the mobile terminal.If the terminal is idle, paging becomes necessary.
Service Request (EPS Bearer Setup) Time, Network Initiated, Control Plane
KPI Category, Object, Type, Unit M Category: Latency, Object: EPS (S-GW S11 interface),
Type: Mean and max (95%) value, Unit: [s]
Definition M It is the time taken by the LTE network to set up an EPS bearer on
request by the P-GW. The EPS bearer has to be created before IP
packets can be sent (DL) to the UE if the UE has no proper EPS bearer
for the given IP packet flow.
The network initiated Service Request Time includes a Paging Time
(ref. to 3.4.1.9 Paging Time) if the UE is idle.
The EPS Bearer Setup procedure is based on the Paging and Service
Request procedures according to References26. [3GPP36.331] and 3.
[3GPP23.401].
Measurement method O Latency measurement, see 2.3.2 Latency Measurements. Lab trial and
stationary users only. Default EPS bearer setup.
The setup times of E-RAB and RB bearers, as well as the Paging time
are part of the EPS setup time and can be calculated from the same
sequence of measurements by taking appropriate traces.
Assumptions, pre-conditions O Conditions for latency measurements see 2.3.2.3 Reference Condi-
tions for Latency Measurements .
UE status before measurement: EMM-REGISTERED , but otherwise
different initial states (idle, dormant or active).
Formula (logical) M Mean value and 95% from all measured samples.Service Request
Message flow, trigger points O The message flow of the Service Request procedure is shown in Figure
6 and Figure 7. The network initiated service request time is measured
between the trigger points BN and EN on the S-GW S11 interface:
• Begin: L2 SCTP: DL DATA NOTIFICATION sent to MME
• End: L2 SCTP: Update Bearer Response sent to MME
If the service request is triggered by the arrival of a DL packet, the inter-
rupt time of the packet stream is measured on the S-GW as the latency
between:
• Begin: arrival of the DL packet at the S-GW toward the idle UE,
• End: forwarding of the same packet to the UE connected.
Related KPIs O Attach Time, Paging Time, E-RAB and RB Setup Times.
Related PM counters O The eNB delay budget of the overall Service Request (ESP) Time is
identical to that of the UE initiated procedures. See related PM counters
and measurements there.
Service Request Time [s] tRRC_Reconfig tRRC_Request–=
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 50/70
50
DN0951948 Issue 01 DRAFT
LTE E2E Field Network Performance - Definitions of Key Performance Indicators
Id:0900d8058072981f
3.4.1.5 Service Request (EPS) Success Rate
Service Request (EPS) Success Rate, Control Plane
KPI Category,Object,
Type, Unit
M Category: Accessibility, Object: EPS (UE R, eNB Uu),
Type: Ratio, Unit: [%]
Definition M This KPI is defined as the ratio between successfully established EPS bearers
compared to the overall number of EPS bearer establishment attempts. It corre-
sponds to the probability that a user or the LTE network can establish an EPS bearer
at any moment in time.
Requests that are terminated by timer expiry (due to the unaccessibility of some LTE
resource) are considered as unsuccessful attempts. Authentication errors (requests
rejected by the MME) are included in the total number of failures.
Only the first RRC CONNECTION REQUEST is to be considered, since counting
retries of the same message would increase the overall number of bearer establish-
ment attempts, and thus reduce the success ratio.
Measurement
method
O Accessibility measurement, see 2.3.3 Service Accessibility Measurements. Field
and lab trial. Stationary users only. For calculating the success ratio, the same series
of "Service Request (EPS) Time" measurements can be used.
Assumptions,
pre-conditions
O Conditions for accessibility measurements see 2.3.3 Service Accessibility Measure-
ments.
See additional conditions in 3.4.1.4 Service Request (EPS) Time, Network Initiated.
Formula (logical) M
Message flow,trigger points
O The message flow diagram is shown in Figure 6 and Figure 7. The trigger points forUE initiated EPS bearer setup success ratio on the R interface are:
• Success: RRC CONNECTION RECONFIGURATION COMPLETE message
sent by the UE, see EUE in Figure 7.
• Total: an attempt is made to send an RRC CONNECTION REQUEST with appro-
priate establishment cause by the UE, see BUE in Figure 6.
Trigger points on the eNB Uu interface:
• Success: RRC CONNECTION RECONFIGURATION COMPLETE message
received by the eNB.
• Total: an attempt is made to receive an RRC CONNECTION SETUP
COMPLETE MESSAGE by the eNB with L3 NAS: SERVICE REQUEST in its
body.
The measurement on the eNB side is an approximation of the success ratio as per-
ceived by the end-user, since it does not include SRB setup failures. The UE side
delivers more precise results.The exact calculation of network initiated EPS bearer
setup success ratios uses the triggers BN and EN, which are also shown in Figure 6
and Figure 7.
EPSSRnumber_of(RRC_CONN_RECONFIGURATION_COMPLETE)
number_of(RRC_CONNECTION_REQUEST)----------------------------------------------------------------------------------------------------------------------------------------------------------------------- 100%⋅=
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 51/70
DN0951948 Issue 01 DRAFT
51
LTE E2E Field Network Performance - Definitions ofKey Performance Indicators
Id:0900d8058072981f
Using online PM measurements on the eNB to count the total number of EPS bearer
setup requests (EPS_SETUP_ATT ) and the successfully terminated ones
(EPS_SETUP_SUCC ), an EPS setup success rate can be calculated (see trigger points
T1 and T2). This ratio is, however, not identical to the user perceived success rate, since
the latter includes additional failure causes, like failures in setting up the SRB signaling
bearer and the ePC part of the EPS bearer, as well as UE authentication errors.
3.4.1.6 Service (EPS Bearer) Drop Rate
Related KPIs O Service Request (EPS) Time, user and network initiated.The "E-RAB Accessibility"
KPI of [3GPP32.450], see References 17. chapter 6.1.1 corresponds well with this
KPI, considering explicitly created "additional" EPS bearers:
For initial EPS bearers, see KPI 3.4.1.2 Attach Success Rate.
Related PM counters O a) 3GPP defines the following measurements for additional EPS bearers:
h. Additional EPS setup attempts: SAEB.NbrAttEstabAdd.QCI
i. Successful setups: SAEB.EstabAddSuccNbr.QCI.
j. Failures: SAEB.EstabAddFailNbr.Cause
For initial EPS bearer setup measurements, see Attach procedure in
3.4.1.2 Attach Success Rate.
b) The corresponding eNB counters are:
• EPS_SETUP_ATT, EPS_SETUP_SUCC, EPS_SETUP_FAIL each counts
both initial and additional EPS setups (i.e. no differentiation is made). See
also trigger points T1 & T2 in Figure 7.
Service (EPS Bearer) Drop Rate, Control Plane
KPI Category, Object, Type, Unit M Category: Availability, Object: EPS (UE R, eNB Uu),
Type: Ratio, Unit: [%]
Definition M It is the ratio between abnormally released bearers and the overall
number of established EPS bearers. An abnormal release is defined as
any EPS bearer termination that was not triggered by the mobile user
(from UE side). Thus, it reflects the probability that an established
bearer is aborted due to insufficient network resources.
Dropping the bearer becomes visible to the end-user if an applicationservice is actively using it. If the application automatically re-estab-
lishes the bearer, it remains unnoticed by the user.
Measurement method O Availability measurement, see 2.3.3 Service Accessibility Measure-
ments. Field and lab trial.
Assumptions, pre-conditions O Conditions for availability measurements see 2.3.3 Service Accessibil-
ity Measurements .The ratio is calculated on the same series of Service
Request (EPS) Time measurements (see 3.4.1.3 Service Request
Time (EPS), UE Initiated and 3.4.1.4 Service Request (EPS) Time,
Network Initiated).
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 52/70
52
DN0951948 Issue 01 DRAFT
LTE E2E Field Network Performance - Definitions of Key Performance Indicators
Id:0900d8058072981f
The EPS bearer release rate determined by online PM counters on the eNB is equiva-
lent to the value of the Service (EPS) Drop Rate KPI.
Formula (logical) M
Message flow, trigger points O Trigger points on UE R / eNB Uu interfaces:
• Success: submission of the message RRC CONNECTION
RECONFIGURATION COMPLETE by the UE, see References26.
[3GPP36.331], or reception by the eNB.
• Drop: successful establishments minus terminations by the user,
i.e. the UE submitting a L3DETACH REQUEST (see References 4.
[3GPP24.008]) carried within the RRC message UL NAS TRANS-
PORT (see References 26. [3GPP36.331]), or the eNB receiving
this message.
Measurements on the eNB side produce equivalent results to measure-
ments on the UE side. The drop ratio calculated this way is valid for
both UE initiated EPS bearers.
For the calculation of a network initiated EPS bearer drop rate, ePC ini-
tiated L3 DETACH REQUEST -s needs to be considered.
Related KPIs O a) Service Request (EPS) Success Rate.
b) The "E-RAB Retainability" KPI of [3GPP32.450], see References
17. in chapter 6.2.1, is an abnormal EPS release rate, which is
related to the above Drop Rate definition, but also considers the
session duration (its unit is [1/s]):
The necessary measurements are defined by 3GPP (see below).
Related PM counters O a) The following online PM counters of the eNB can be used to calcu-
late the Service Drop Rate:
EPS_SETUP_SUCC, EPS_REL_EPC_INI, EPS_REL_ENB_INI.
b) 3GPP defines the following measurements for an abnormal EPS
bearer release: SAEB.RelAct.QCI, SAEB.SessionTimeUE and
SAEB.SessionTimeQCI.QCI
EPSBearerD Rnumber_of(dropped_calls)
number_of(successful_calls)------------------------------------------------------------------------------ 100%⋅=
R2
SAEB.RelAct.[QCI]
QC I
∑
SAEB.SessionTimeUE
QC I
∑-----------------------------------------------------------------------=
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 53/70
DN0951948 Issue 01 DRAFT
53
LTE E2E Field Network Performance - Definitions ofKey Performance Indicators
Id:0900d8058072981f
3.4.1.7 Handover Procedure Time
Handover Procedure Time, Control Plane
KPI Category, Object, Type, Unit M Category: Mobility (Latency), Object: LTE (UE R interface),
Type: Mean and max (95%) value, Unit: [ms]
Definition M It denotes the total time needed for the hand-over procedure as seen
by the UE. It begins by receiving a Handover Command from the SeNB
and ends by sending the Handover Confirm response to the TeNB by
the UE. Its relevance is the discontinuity of the IP packet flow in the user
plane that is implied by it (also called service interruption).
The value of the HO Procedure Time KPI depends on the hand-over
scenario. The following HO scenarios are distinguished (though not
directly seen by the UE):
•intra LTE intra frequency mobility
• intra LTE inter frequency mobility
• inter RAT mobility (LTE ↔ 2G/3G,
in this release only LTE → 3G/UMTS)
• intra vs. inter eNB
• intra vs. inter MME/S-GW
Remark: The value of the HO Procedure Time itself does not depend
on the direction of an ongoing data transfer (UL, or DL). It is the LTE
Service Interrupt Time (HO) which will be different.
Measurement method O Mobility (HO) measurement, see 2.3.5 Mobility (Handover) Measure-
ments. Lab and field trials, mobile user scenarios only, crossing of cell
borders included. The handover time should be measured on the UE R
interface. However, measurements on the source eNBs also provide
good approximations of the handover time as perceived by the UE.
They even have some advantage, since the SeNB sees the full
handover procedure, the handover preparation phase included.
End-user delays can be measured with test equipment only, since ter-
minals do not indicate cell changes toward the end-user.
Assumptions, pre-conditions O Conditions for handover measurements see 2.3.5 Mobility (Handover)
Measurements .
UE status before measurement: registered and connected.
Formula (logical) M Mean value and 95% from all measured samples.
Handover Procedure Time [ms] tHO_Confirm tHO_Command–=
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 54/70
54
DN0951948 Issue 01 DRAFT
LTE E2E Field Network Performance - Definitions of Key Performance Indicators
Id:0900d8058072981f
Message flow, trigger points O The HO procedure time is seen by the UE as the elapsed time between
(see message flow in Figure 8):
• Start: UE receives Handover Command from source eNB (SeNB).
• Stop: UE sends Handover Confirm to the target eNB (TeNB).
It is seen on the SeNB as the elapsed time between:
• Start: SeNB sends Handover Request to TeNB (X2).
• Stop: SeNB receives UE Context Release from TeNB.
It is seen on the TeNB as the elapsed time between:
• Start: target eNB receives Handover Request from SeNB (X2).
• Stop: target eNB sends UE Context Release to source eNB.
Related KPIs O a) (LTE) Service Interrupt Time, (RB) Service Interrupt Time, HO
Success Rate.
b) No 3GPP defined KPI for HO procedure time found.
Related PM counters O HO Procedure Time related measurements are not defined in 3GPP
Release 8 documents.
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 55/70
DN0951948 Issue 01 DRAFT
55
LTE E2E Field Network Performance - Definitions ofKey Performance Indicators
Id:0900d8058072981f
Figure 8 Handover Procedure
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 56/70
56
DN0951948 Issue 01 DRAFT
LTE E2E Field Network Performance - Definitions of Key Performance Indicators
Id:0900d8058072981f
3.4.1.8 Handover Success Rate
Handover Procedure Time, Control Plane
KPI Category, Object,
Type, Unit
M Category: Mobility (Accessibility), Object: eNB (X2 interfaces),
Type: Ratio, Unit: [%]
Definition M The Handover Success Rate is the ratio between successfully executed (commit-
ted) HO procedures and the number of all Handover attempts.
Measurement method O Mobility (HO) measurement, see 2.3.5 Mobility (Handover) Measurements. Field
and lab trial.
For calculating the success ratio, the same series of "Service Interrupt Time" mea-
surements can be used. For restrictions in measurement scenarios, refer to the
description of the KPI "(LTE) Service Interrupt Time".
The measurement is to be executed for different HO scenarios (see description
with the KPI 3.4.1.7 Handover Procedure Time).
If practical limitations make it difficult carry out a sufficient number of performance
tests with terminals, eNB online statistics can be used as approximations (as
described below). In case of Inter-RAT scenarios, the eNB statistics cover the LTE
leg of the hand-off only. HO failures in the 2G/3G leg have to be collected with cor-
responding statistics of the BTS/NodeB.
Assumptions,
pre-conditions
O Conditions for mobility measurements see 2.3.5 Mobility (Handover) Measure-
ments.
See additional conditions at the KPI “(LTE) Service Interrupt Time".
Formula (logical) M
Message flow,
trigger points
O The success rate perceived by end-users is approximated with the success rate
as seen on the source eNB. Trigger points on the eNB X2 interface:
• Total #: Handover Request sent by SeNB to TeNB (X2).
• Success #: UE Context Release message received by SeNB from TeNB.
In case of Intra eNB HO procedures, no Handover Request is sent, both trigger
points are to be counted by the (Source) eNB internally.
Related KPIs O a) Handover Procedure Time
b) “E-UTRAN Mobility Success Rate" in [3GPP32.450], see References17.
Chapter 6.5.1, is a success rate, which applies to all Intra- and Inter-RAT
mobility scenarios:
The KPI is calculated by multiplying the success rates for HO preparation and
execution phases. The relationship of variables in this expression and the HO
measurements defined in [3GPP32.425],see References 15., is still to be clar-
ified.
HOSRnumber_of(Handover_Confirm)
number_of(Handover_Request)-------------------------------------------------------------------------------------- 100%⋅=
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 57/70
DN0951948 Issue 01 DRAFT
57
LTE E2E Field Network Performance - Definitions ofKey Performance Indicators
Id:0900d8058072981f
The calculation of HO Success Rate does not differentiate between different causes of
a negative hand-over decision. All rejected HO attempts are considered as unsuccessful
handovers.
Related PM counters O a) 3GPP defines measurements / counters on eNB to support HO success ratio
calculations (see References 15. [3GPP32.425] chapter 4.3). Examples for
selected scenarios:
Intra-RAT, intra-eNB handovers: • # HO attempts: HO.IntraENBOutAtt.Cause
• # HO success: HO.IntraENBOutSucc.Cause
where Cause identifies the cause for handover
Intra-RAT, inter-eNB handovers:
• # HO attempts: HO.InterENBOutPrepAtt
• # HO attempts: HO.InterENBOutAtt.Cause
• # HO success: HO.InterENBOutSucc.Cause
Handover measurements on neighbor cell basis:
• # HO attempts: HO.OutAttTarget.Cause
• # HO success: HO.OutSuccTarget.Cause
Inter-RAT handovers (LTE -> 2G/3G):
• # HO attempts: HO.IartOutAtt.Cause
• # HO success: HO.IartOutSucc.Cause
b) LTE product related counters on eNB which support the calculation of
handover success rates:
Intra-RAT, intra-eNB handovers:
• # HO attempts: INTRA_HO_PREP
• # HO success: INTRA_HO_SUCC
Intra-RAT, iner-eNB handovers:
• # HO attempts: INTER_HO_PREP
• # HO success: INTER_HO_SUCC
Inter-RAT handovers (LTE -> 2G/3G):
• # HO attempts: INTER_RAT_HO_PREP
• # HO success: INTER-RAT_HO_SUCC
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 58/70
58
DN0951948 Issue 01 DRAFT
LTE E2E Field Network Performance - Definitions of Key Performance Indicators
Id:0900d8058072981f
3.4.1.9 Paging Time
Paging Time, Control Plane
KPI Category, Object, Type, Unit M Category: Latency, Object: LTE (MME, S-GW; S11, S1-U),
Type: Mean and max (95%) value, Unit: [s]
Definition M Paging is initiated :
• By the arrival of a DL packet at the S-GW for an UE whose location
is not known, i.e. its location information does not exist in the S-GW,
or
• By network initiated EPS bearer setup requests (ISR flag is set)
It denotes the total procedure time from starting the paging request in
DL and terminating it with the subsequent service request (EPS bearer
setup) of the UE after it has been located. Thus, the paging time is per
definition the difference between network and UE initiated ServiceRequest Times.
Trigger events can be defined on the MME and S-GW as initiators of
the paging request, either on the S11 (MME - S-GW) or on the S1-U
(UE - S-GW) interfaces, see "Trigger points" below.
Measurement method O Latency measurement, see 2.3.2 Latency Measurements. Lab trial with
mobile user scenario only, crossing of cell borders included.
The paging time can be measured on the S11 (MME - S-GW), or S1-U
(UE - S-GW) interfaces. End-user delays cannot be measured, since
paging is always initiated by the network.
Assumptions, pre-conditions O Conditions for latency measurements see 2.3.2.3 Reference Condi-
tions for Latency Measurements.
UE status before measurement: registered, but idle (not connected).
Formula (logical) M The paging time is calculated as the difference between network initi-
ated service request and UE initiated service request times.
Message flow, trigger points O See message flow in Figure 9.The paging time can be approximated on
the MME S11 with:
• Start: L2 SCTP: DL Data Notification received for an UE
• Stop: L3 NAS: SERVICE REQUEST received from the same UE
On S-GW S1-U, latency is measured between:
• Start: DL packet arrives for an idle UE
• Stop: DL packet is forwarded to the connected UE
The latter latency is a good metric of DL "service interrupt time" for an
IP packet stream caused by paging. The paging time has to be calcu-
lated as the measured latency minus the UE initiated Service Request
Time (EPS).
Related KPIs O a) Service Request Time (EPS), UE and network initiated.
b) No 3GPP Release 8 defined KPIs for Paging Time found.
Pagi ng Time [s] tSRT network initiated tSR T UE initiated–=
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 59/70
DN0951948 Issue 01 DRAFT
59
LTE E2E Field Network Performance - Definitions ofKey Performance Indicators
Id:0900d8058072981f
Figure 9 Paging Procedure
Related PM counters O Paging Time related measurements are neither defined in 3GPP
Release 8, nor in LTE reference documents.
LTE-Uu S1- MME S11 S6a
UE eNodeB MME S-GW / P-GW HSS
UE EMM-REGISTERED
and ECM-IDLE
DL Packet
L2 SCTP: DL Data
Notification
RRC: DL INFORMATION
TRANSFER
(L3 NAS: PAGING)
D
e
l
ay
L2 SCTP: DL Data
Notification Ack
Paging Procedure
[3GPP TS 36.413]
S1-AP: DL NAS
TRANSPORT
(L3 NAS: PAGING)
To all cells of all TAs
UE in EMM-REGISTERED
and ECM-CONNECTED
UE Triggered Service
Request Procedure
[3GPP TS 23.401]
Forward DL Packet to UE
L2 SCTP: Stop Paging
BN
EN
T1
T2L2 SCTP: DL Data
Notification Reject
Retry (optionally)
T
i
m
e
r
Paging Failure
Paging Attempt
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 60/70
60
DN0951948 Issue 01 DRAFT
LTE E2E Field Network Performance - Definitions of Key Performance Indicators
Id:0900d8058072981f
3.4.1.10 Paging Failure Rate
Paging Failure Rate, Control Plane
KPI Category, Object, Type, Unit M Category: Accessibility, Object: MME (S1-AP, S11),
Type: Ratio, Unit: [%]
Definition M The Paging Failure Rate is the ratio between unsuccessful paging
requests and the number of all paging attempts initiated by the MME.
Retries of the same paging request by the MME are not counted as new
attempts. Similarly, multicasting the same request to more than one
eNBs (in the UE´s tracking areas) is considered as one attempt.
Measurement method O Accessibility measurement, see 2.3.3 Service Accessibility Measure-
ments. Field and lab trials.
For calculating the failure ratio, the same series of "Service Request
(EPS) Success Rate" measurements can be used, however, restricted
to the network initiated bearer setup scenario. For restrictions in mea-
surement scenarios, see also the KPI “Service Request Success Rate".
The existence of online performance counters for paging on the MME
can help to avoid the execution of performance tests repeatedly.
Suitable counters are indicated in the message flow in Figure 9 (trigger
points T1 and T2).
Assumptions, pre-conditions O Conditions for accessibility measurements, see 2.3.3 Service Accessi-
bility Measurements.
See additional conditions at the KPI "Service Request Success Rate".
Formula (logical) M
Message flow, trigger points O Trigger points on MME (see T1: Attempt and T2: Failure on the Figure
9):
• Total #: different L3 NAS Paging Requests sent to an eNB (S1-AP).
• Failure #: DL Data Notification Reject messages sent to S-GW
(S11)
Related KPIs O a) Paging Time, Service Request Success Rate - network initiated.
b) No Paging related KPI is defined in 3GPP Release 8 documents.
Related PM counters O Directly applicable trigger events are T1 and T2 as shown in Figure 9.
a) 3GPP defined counter (see References15. [3GPP32.425] Chapter
4.7) on eNB is:
• Discarded Paging Records: PAG.DiscardedNbr
b) LTE product related counters defined for eNB in:
• Requests: RRC_PAGING
• Responses: SIG_CON_EST_ATT.Cause = PagingResponse
Using these counters, the Paging Failure Rate can be calculated as:
Paging FR =( 1 - SIG_CON_EST_ATT.Cause = PagingResponse /
RRC_PAGING *100%)
PagingFR number_of(Paging_Failures)number_of(Paging_Attempts)-------------------------------------------------------------------------------- 100%⋅=
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 61/70
DN0951948 Issue 01 DRAFT
61
LTE E2E Field Network Performance - Definitions ofKey Performance Indicators
Id:0900d8058072981f
3.4.1.11 (LTE) Round Trip Time
Round Trip Time (RTT), User Plane
KPI Category, Object, Type, Unit M Category: Latency, Object: LTE (UE R, P-GW Gi interfaces),
Type: min (CDF 5%), mean (CDF 50%), max (CDF 95%) values,
Unit: [ms]
Definition M RTT in UL is the interval between sending a datagram by the UE &
receiving the corresponding reply from an IP peer entity connected to
the Gi interface of the P-GW.
RTT in DL is the interval between sending a datagram to the UE &
receiving the corresponding reply by the IP host (peer entity).
Measurement method O Latency measurement, 2.3.2 Latency Measurements. Field and lab
trials. Stationary users in different cell positions, which are uniformly
distributed across the cell.
RTT is measured with the Ping application between the UE and an IP
host (peer entity). As average RTT figure, the output of the Ping appli-
cation is used (e.g. "MS-DOS >ping -n count -l size host_name"). Size
of ICMP packets: 32, …, MTU, 2x MTU.
Ping should be executed at least 100 times to reduce the impact of the
first ICMP message, which triggers the setup of radio bearer establish-
ments in UL and DL directions.
Assumptions, pre-conditions O Conditions for latency measurements see 2.3.2.3 Reference Condi-
tions for Latency Measurements.
UE status before measurement: registered and connected.
The IP peer entity should be located as close as possible to the SGi
interface. Additional delays between the P-GW and the IP host should
be avoided or minimized.
Formula (logical) M Mean value and 95% from all measured samples.
Message flow, trigger points O Trigger points both from UE, and from IP peer entity view:
• Start: ICMP ECHO REQUEST
• Stop: ICMP ECHO REPLY
Related KPIs O a) Network Delay UL / DL KPI for LTE, RB and IP based transport
bearers X2, S1, S5/S8)
b) No 3GPP defined KPI for LTE RTT found.
c) See "{Service} Round Trip Time" KPI definition for Ping service in
[ETSI102.250-2], see References 39. Chapter 4.6.3.6.
Related PM counters O Measurements of round trip times are neither defined in 3GPP nor in
LTE reference documents.
Round Trip Time [ms] tICMP Echo Reply tIC MP Echo Request–=
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 62/70
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 63/70
DN0951948 Issue 01 DRAFT
63
LTE E2E Field Network Performance - Definitions ofKey Performance Indicators
Id:0900d8058072981f
(~minutes). Of equal interest are peak values observed for a short time period only
(~seconds).
Please note that data rate KPIs have also been defined for application services, e.g.
User Data Rate KPI of the Data Download / Upload service, but these do not character-
ize the capability of the LTE network alone, but also depend on the application service
and its protocols.
3.4.1.13 (LTE) Packet Loss Rate
(LTE) Packet Loss Rate UL / DL, User Plane
KPI Category, Object, Type, Unit M Category: Availability, Object: LTE (UE R interface, IP peer entity)
Type: Ratio, Min (CDF 5%), Mean (CDF 50%) Max (CDF 95%) values
Unit: [%]
Definition M This is the ratio between the numbers of lost or corrupted IP packets,and of all IP packets sent. Corrupted IP packets are those that contain
bit errors in their headers or in their payload.Packets with "residual", i.e.
undetected errors are not counted as lost.
Measurement method O Availability measurement, see 2.3.3 Service Accessibility Measure-
ments.
The EPS bearer is to be configured such that corrupted IP packets are
not delivered to the application.
Measured in loaded and unloaded network, under different radio condi-
tions. Stationary and mobile users.
The packet loss rate should be measured in test scenarios, where han-
dovers occur. The number of handovers should be oriented toward theNSN reference traffic model, or, in operator trials, toward the operator´s
traffic model.
Assumptions, pre-conditions O Conditions for availability measurements, see 2.3.3 Service Accessibil-
ity Measurements.
Measured in loaded and unloaded network, under different radio condi-
tions. Stationary and mobile user including HO scenarios.
The ratio can be calculated on a series of User Data Rate (see
2.3.1.2 Peak User Data Rate) measurements.
Formula (logical) M
Message flow, trigger points O UL / DL:
• Total: packages sent at UE R if / IP peer entity
• Lost: Total minus packets received at IP peer entity / UE R if.
Related KPIs O a) (RB) Initial Radio Block Error Rate (BLER)
b) No PLR KPI (LTE level) is defined in 3GPP and ETSI documents.
Related PM counters O Measurements of packet loss rate on LTE level (R <-> Gi) are neither
defined in 3GPP nor in LTE reference documents.
PLRnumber_of(lost_corrupted_packets)
number_of(all_packet_sent)------------------------------------------------------------------------------------------------ 100%⋅=
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 64/70
64
DN0951948 Issue 01 DRAFT
LTE E2E Field Network Performance - Definitions of Key Performance Indicators
Id:0900d8058072981f
3.4.2 Radio Bearer KPIs
3.4.2.1 (RB) Packet Loss Rate
(RB) Packet Loss Rate UL / DL, User Plane
KPI Category, Object, Type, Unit M Category: Availability, Object: RB (UE R interface, eNB)
Type: Ratio, Min (CDF 5%), Mean (CDF 50%) Max (CDF 95%) values
Unit: [%]
Definition M This is the ratio between the numbers of lost or corrupted IP packets,
and of all IP packets sent. Corrupted IP packets are those that contain
bit errors in their headers or in their payload.
Packets with "residual", i.e. undetected errors are not counted as lost.
Measurement method O The KPI is not measured on its own. Traces are to be taken on the UE
and eNB during (LTE) Packet Loss Rate measurements, see3.4.1.13 (LTE) Packet Loss Rate. Lab trials with results for internal use.
Assumptions, pre-conditions O See comment above.
Formula (logical) M
Message flow, trigger points O UL / DL:
• Total #: packages sent on UE R / eNB
• Lost #: Total minus packets received on eNB / UE R
Measured at the upper SAPs of the L2 PDCP.
Related KPIs O a) Initial Radio Block Error Rate (BLER)
b) No PLR KPI (RB level) is defined in 3GPP and ETSI documents.
Related PM counters O a) The following measurements are defined in LTE for the number of
discarded (dropped) PDUs on the eNB in DL:
• Discarded PDCP SDUs: PDCP_SDU_DISCARD
• Discarded RLC blocks: SDU_DISCARD_DL_DTCH
b) The following measurements are defined by 3GPP in
[3GPP32.425], see References 15. chapters 4.4.3-4, for packet
(PDCP SDU) loss and drop rates:
• Drop rate on eNB in DL: DRB.PdcpSduDropRateDl.QCI
• Loss rate on Uu in DL: DRB.PdcpSduAirLossRateDl.QCI
• Loss rate on Uu in UL: DRB.PdcpSduLossRateUl.QCI
A packet drop rate on eNB in UL is not defined.The loss rate values
may refer to the L2 protocols PDCP, RLC or MAC (see
[3GPP32.314] chapter 4.1.5.1-3)
RBPLRnumber_of(lost_corrupted_packets)
number_of(all_packet_sent)------------------------------------------------------------------------------------------------ 100%⋅=
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 65/70
DN0951948 Issue 01 DRAFT
65
LTE E2E Field Network Performance - Definitions ofKey Performance Indicators
Id:0900d8058072981f
3.4.2.2 (RB) User Data Rate
(RB) User Data Rate UL / DL, User Plane
KPI Category, Object, Type, Unit M Category: Throughput, Object: RB, eNB,
Type: Min (CDF 5%), Mean (CDF 50%), Max (CDF 95%) values,
Unit: [Mbps]
Definition M The metric describes the UDP/IP data rate achievable by one user. It
can be given as single user data rate if only one user is active in the
cell, or as multi-user data rate for a given number of concurrently active
users.
The user data rate distribution over the cell is given as function (CDF)
of the radio conditions characterized by the SINR.
The time variation of the KPI value in a given cell position is given with
its maximum (95%) (also referred to as instantaneous "Peak User Data
Rate"), and with its average (mean) value.The Peak User Data Rate corresponds to the Cell Throughput KPI on
UDP/IP level, when the number of active users in the cell is n=1 and
best radio conditions are observed.
The User Data Rate is also called "User Throughput" in the literature.
Measurement method O Throughput measurement, see 2.3.1 Throughput Measurements. Sta-
tionary and mobile user. Cell positions: best, medium, cell edge, includ-
ing different HO scenarios. Field and lab trials.
The eNB assigns all resources (radio blocks) to this user, except for the
radio blocks used for signaling and controlling. Measured on UDP/IP
level for all UE categories (1-5). With the help of trace analysis, PDCP,
RLC, MAC and PHY layer throughputs can also be defined.With and w/o concurrency in UL / DL traffic.
The mean user data rate is calculated over a few minutes, peak data
rate values are averaged over short periods of time (e.g. 1s).
Assumptions, pre-conditions O Conditions for capacity measurements, see 2.3.1.2 Peak User Data
Rate.
All transport bearers between the UE and IP peer entity should have
higher capacity than the radio link in order to avoid bandwidth bottle-
necks.
Formula (logical) M
Message flow, trigger points O Trigger events on the eNB:
• UL: eNB receiving PDUs
• DL: eNB sending PDUs
Measured at the upper SAPs of the protocols PDCP, RLC, MAC or
PHY.
Related KPIs O a) (LTE) User Data Rate, Cell Throughput.
b) See 3GPP defined KPIs in 3.4.2.3 Cell Throughput.
RBUserDataRate transferred_data_volume [bytes] 8⋅t ransfer_time [s] 10
6–⋅
----------------------------------------------------------------------------------------------------=
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 66/70
66
DN0951948 Issue 01 DRAFT
LTE E2E Field Network Performance - Definitions of Key Performance Indicators
Id:0900d8058072981f
3.4.2.3 Cell Throughput
Related PM counters O See 3GPP defined measurements with the KPI in 3.4.2.3 Cell Through-
put.
Cell Throughput UL / DL, User Plane
KPI Category, Object, Type, Unit M Category: Throughput, Object: UE, RB Uu,
Type: Min (CDF 5%), Mean (CDF 50%), Max (CDF 95%) values,
Unit: [Mbps]
Definition M The metric shows the sustainable aggregate throughput of the cell (in
UL/DL) available to "n" stationary users distributed uniformly in the cell
and running a typical mix of applications. The "cell throughput" is the
sum of all bits transported in all radio blocks carrying PDUs (i.e. bits in
UL-SCH / DL-SCH transport blocks) during one second.The cell capacity is also given as peak value (called peak cell capacity,
or throughput), which is defined as the aggregate throughput of "n"
users all located in best radio conditions.
The cell throughput value is defined here on PHY level, but could be
given for other protocol levels (UDP/IP, PDCP, RLC, MAC), too. When
the (peak, average) cell throughput is expressed on UDP/IP level, it cor-
responds to the (peak, mean) user data rate value at comparable radio
conditions.
Measurement method O Throughput measurement, 2.3.1 Throughput Measurements. Station-
ary users uniformly distributed in the cell. UE categories according to
application mix. Peak value measured with one user in LoS cell positionusing UDP/IP load to approach the full buffer condition.
Field trial without target value (only to learn the available capacity).
Measured with UDP/IP traffic on PHY layer (UL-SCH / DL-SCH trans-
port block bits) over several minutes. With the help of trace analysis
PDCP, RLC and MAC layer values can also be given.
Assumptions, pre-conditions O Conditions for throughput measurements see 3.4.1.2 Attach Success
Rate.
All IP transport bearers (S1, S5/S8, SGi) between the UE and IP peer
entity should have higher bandwidth than the cell capacity to avoid
bandwidth bottlenecks.
Formula (logical) M
Message flow, trigger points O Trigger points on the eNB Uu interface:
• UL: eNB receiving PDUs / RLC blocks / transport data blocks
• DL: eNB sending PDUs / RLC blocks / transport data blocks
Related KPIs O a) (RB) User Data Rate
b) 3GPP defines the KPI "E-UTRAN IP Throughput" in [3GPP32.450],
see References 17. chapter 6.3.1.
CellThroughputtransferred_data_volume [bytes] 8⋅
transfer_time [s] 106–
⋅
----------------------------------------------------------------------------------------------------=
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 67/70
DN0951948 Issue 01 DRAFT
67
LTE E2E Field Network Performance - Definitions ofKey Performance Indicators
Id:0900d8058072981f
Related PM counters O a) The following measurements are available on eNB in LTE refer-
ence document, which are related to the Cell Throughput KPI:
• CELL_THROUGHPUT_PDCP (separate counters for min,
mean and peak values, both for UL and DL on Uu interface) • PDCP_SDU (same for one cell)
b) The following measurements are defined by 3GPP in
[3GPP32.425] Chapters 4.4.1 for cell data (PDCP SDU) rates:
• Average data rate in DL: DRB.PdcpSduBitrateDl.QCI
• Maximal data rate in DL: DRB.PdcpSduBitrateDlMax
• Average data rate in UL: DRB.PdcpSduBitrateUl.QCI
• Maximal data rate in UL: DRB.PdcpSduBitrateUlMax
The data rate values are measured on the eNB at the upper SAP of L2
PDCP and refer to SDU bits. SDUs that are forwarded over X2/S1 to
another eNB are subtracted from the overall bit count.
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 68/70
68
DN0951948 Issue 01 DRAFT
LTE E2E Field Network Performance - Definitions of Key Performance Indicators
Id:0900d8058072981f
3.4.2.4 (RB) Service Interrupt Time (HO)
(RB) Service Interrupt Time (HO), User Plane
KPI Category, Object, Type, Unit M Category: Latency, Object: RB (UE, eNB Uu),
Type: Mean and max (95%) value, Unit: [ms]
Definition M The Service Interrupt Time is the interval between the last
sent/received RLC block carrying user PDU of a continuous UL/DL data
stream in the old cell and the first sent/received user PDU in the new
cell measured on the RLC level (user plane break).
The value of the KPI depends on the handover scenario. See HO sce-
narios at the KPI 3.4.1.7 Handover Procedure Time.
Measurement method O Mobility (HO) measurement, see 2.3.5 Mobility (Handover) Measure-
ments.
Only mobile user scenarios including crossing of cell borders. The test
application is an FTP upload/download data transfer, or real-time
media flow during the handover.
Assumptions, pre-conditions O Conditions for mobility measurements see 2.3.5 Mobility (Handover)
Measurements.
It is assumed that the X2 interface is enabled between the source and
target eNBs (in inter eNB HO scenarios).
Formula (logical) M
Mean value and 95% from all measured samples.
Message flow, trigger points O The trigger points for service interrupt time (from UE point of view):
• Start: UE sends the last packet to the source eNB
• Stop: UE sends the first packet to the target eNB.
See Figure 8.
Related KPIs O a) (LTE) Service Interrupt Time (HO), Voice Interrupt Time (HO)
b) No related KPI definitions found in 3GPP Rel. 8 documents.
Related PM counters O Measurements of service interrupt times caused by HO are neither
defined in 3GPP, nor in LTE reference documents.
Service Interrupt Time [ms] t first packet TeNB tlas t packet SeNB–=
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 69/70
DN0951948 Issue 01 DRAFT
69
LTE E2E Field Network Performance - Definitions ofKey Performance Indicators
References
Id:0900d8058072b1b0
4 References
1. [3GPP21.905] 3GPP TR 21.905; v 8.x.x; Vocabulary for 3GPP Specification
2. [3GPP23.203] 3GPP TS 23.203; V8.7.0 (2009-09); Policy and charging controlarchitecture
3. [3GPP23.401] 3GPP TR 23.401; v.8.x.x; General Packet Radio Service (GPRS)
enhancements for Long Term Evolution (LTE) access
4. [3GPP24.008] 3GPP TR 24.008; V8.5.0 (2009-03); Mobile Radio Interface Layer 3
Specification; Core network protocols; Stage 3
5. [3GPP24.228] 3GPP TR 24.228; V5.15.0 (2006-10); Signaling flows for the IP mul-
timedia call control based on Session Initiation Protocol (SIP) and Session Descrip-
tion Protocol (SDP);Stage 3 (Release 5)
6. [3GPP24.229] 3GPP TR 24.229; V8.8.0 (2009-06);IP multimedia call control
protocol based on Session Initiation Protocol (SIP) and Session Description
Protocol (SDP);Stage 3 (Release 8)7. [3GPP25.913] 3GPP TR 25.913; V8.0.0 (2008-12); Requirements for Evolved
UTRA (E-UTRA) and Evolved UTRAN (E-UTRAN)
8. [3GPP25.943] 3GPP TR 25.943; V8.0.0 (2008-12); Deployment Aspects (Release
8)
9. [3GPP26.114] 3GPP TS 26.114 V8.2.1 (2009-03); IP Multimedia Subsystem (IMS);
Multimedia Telephony; Media handling and interaction (Release 8)
10. [3GPP29.002] 3GPP TS 29.002, V8.x.x; Mobile Application Part (MAP)
11. [3GPP29.060] 3GPP TS 29.060; v 8.x.x; General Packet Radio Service (GPRS);
GPRS Tunneling Protocol (GTP) across the Gn and Gp interface
12. [3GPP32.410] 3GPP TS 32.410; V8.0.0 (2009-03); Telecommunication manage-
ment; Key Performance Indicators for UMTS and GSM (Release 8)13. [3GPP32.421] 3GPP TS 32.421, V8.4.0 (2008-12); Telecommunication manage-
ment; Subscriber and equipment trace; Trace concepts and requirements
14. [3GPP32.423] 3GPP TS 32.423, V8.0.0 (2009-03); Telecommunication manage-
ment; Subscriber and equipment trace, Trace data definition & management
15. [3GPP32.425] 3GPP TS 32.425, V8.1.0 (2009-06); Telecommunication manage-
ment; PM Performance Measurements, E-UTRAN (Release 8)
16. [3GPP32.426] 3GPP TS 32.426, V8.0.0 (2009-03); Telecommunication manage-
ment; Performance Management (PM); Performance measurements, Evolved
Packet Core network (EPC)
17. [3GPP32.450] 3GPP TS 32.450, V8.1.0 (2009-06); Telecommunication manage-
ment; Key Performance Indicators for E-UTRAN: Definitions (Release 8)18. [3GPP32.451] 3GPP TS 32.451, V8.0.0 (2009-03); Telecommunication manage-
ment; Key Performance Indicators for E-UTRAN: Requirements (Release 8)
19. [3GPP32.814] 3GPP TS 32.814, V7.0.0 (2007-03); Telecommunication manage-
ment; UTRAN and GERAN Key Performance Indicators (KPI), (Release 7)
20. [3GPP36.300] TS 36.300; v 8.x.x; Evolved Universal Terrestrial Radio Access (E-
UTRA) and Evolved Universal Terrestrial Radio Access Network (E-UTRAN);
Overall description; Stage 2
21. [3GPP36.306] TS 36.306; V8.3.0 (2009-03); Evolved Universal Terrestrial Radio
Access (E-UTRA); User Equipment (UE) Radio Access Capabilities
22. [3GPP36.211] TS 36.211; V8.7.0 (2009-05); Evolved Universal Terrestrial Radio
Access (E-UTRA); Physical Channels and Modulation
8/13/2019 LTE E2E Field Network Performance - KPI Definitions RL10, Customer Version
http://slidepdf.com/reader/full/lte-e2e-field-network-performance-kpi-definitions-rl10-customer-version 70/70
LTE E2E Field Network Performance - Definitions of Key Performance Indicators
References
23. [3GPP36.212] TS 36.212; V8.7.0 (2009-05); Evolved Universal Terrestrial Radio
Access (E-UTRA); Multiplexing and channel coding
24. [3GPP36.214] TS 36.214; V8.6.0 (2009-03); Evolved Universal Terrestrial Radio
Access (E-UTRA); Physical layer - Measurements25. [3GPP36.314] TS 36.314; V8.1.0 (2009-03); Evolved Universal Terrestrial Radio
Access (E-UTRA); Layer 2 - Measurements
26. [3GPP36.331] TS 36.331; V8.6.0 (2009-06); Evolved Universal Terrestrial Radio
Access (E-UTRA); Radio Resource Control (RRC) protocol specification
27. [3GPP36.401] TS 36.401; v 8.x.x; Evolved Universal Terrestrial Radio Access
Network (E-UTRAN); Architecture description
28. [3GPP36.413] TS 36.413; V8.6.1 (2009-06); Evolved Universal Terrestrial Access
Network (E-UTRAN) ; S1 Application Protocol (S1 AP)
29. [3GPP36.423] TS 36.423; v 8.x.x; Evolved Universal Terrestrial Radio Access
Network (E-UTRAN); X2 Application Protocol (X2AP)
30. [3GPP36.521-1] 3GPP TS 36.521-1 V8.2.1 (2009-06); Evolved Universal TerrestrialRadio Access (E-UTRA) User Equipment (UE) conformance specification, Radio
transmission and reception, Part 1: Conformance Testing
31. [3GPP36.942] TS 36.942; V8.2.0 (2009-05); Evolved Universal Terrestrial Radio
Access (E-UTRA); Radio Frequency (RF) system scenarios
32. [RFC792] IETF RFC 792: Internet Control Message Protocol
33. [RFC979] IETF RCF 979: File Transfer Protocol
34. [RFC2679] IETF RCF 2679: A One-way Delay Metric for IPPM, September 1999
35. [RFC2680] IETF RCF 2680: A One-way Packet Loss Metric for IPPM, September
1999
36. [RFC2681] IETF RCF 2681, A Round-trip Delay Metric for IPPM, September 1999
37. [RFC3261] IETF RCF 3261, SIP - Session Initiation Protocol, June 2002
38. [RFC4960] IETF RFC 4960, Stream Control Transmission Protocol
39. [ETSI102.250-2] ETSI TS 102 250-2 V1.3.1 (2005-07); Speech Processing, Trans-
mission and Quality Aspects (STQ); Part 2: Definition of Quality of Service parame-
ters and their computation
40. [ITUE500] ITU-T Recommendation E.500, Traffic intensity measurement principles,
11/98
41. [ITUE800] ITU-T Recommendation E.800, Definitions of terms related to quality of
service, 09/2008
42. [ITUG107] ITU-T Recommendation G.107 (03/2003), The E-model, a computational
model for use in transmission planning