Table of Contents........................................................................................................................................ 3
List of Figures.............................................................................................................................................. 6
List of Tables............................................................................................................................................... 7
2.3 Field and Network KPIs.............................................................................................................11
3 Field KPIs........................................................................................................................................... 11
3.1 Field KPI Definitions................................................................................................................... 11
3.1.1 (FTP) Service Accessibility Ratio...........................................................................................12
TD-LTE KPI Optimization Guidelines for RL15TD Version: 1.1
List of Tables
Table 1 List of Field KPIs........................................................................................................................... 12
Table 2 List of Network KPIs..................................................................................................................... 33
Table 3 Baseline Configuration of CMCC Large Scale Trial in Hangzhou.................................................34
Table 4 Field Results of CMCC Large Scale Trial in Hangzhou................................................................35
Table 5 Baseline Configuration of Bharti Airtel TD-LTE Trial.....................................................................39
Table 6 Baseline Configuration of Maxis Aircel TD-LTE Trial....................................................................43
Table 7 Baseline Configuration of Voentelecom TD-LTE Trial..................................................................46
TD-LTE KPI Optimization Guidelines for RL15TD Version: 1.1
1 Introduction
This document is focused on what kind of KPI values can be achieved in TD-LTE with certain network configurations by collecting KPIs from trial networks and commercial networks. It can be seen as a complementary to [3], in which the KPI targets mainly come from lab tests and/or system-level simulations. KPIs from this document can be taken as the baseline KPIs which are useful for KPI negotiation with our customers and for KPI optimization.
This document doesn’t actually contain any approaches or methods which can be used in network optimization phase. Please refer to [5] for optimization guidelines for TDDFDD- LTE which is a counterpart of the FDD. Unfortunately, there is no official version for TDD-LTE version [6]yet.
KPIs in this document are specified for RL15TD.
This document is organized as follow:
Chapter 1: This chapter gives an introduction to main purpose of this document.
Chapter 2: This chapter gives an overview of KPIs in LTE, including KPI categories, KPI hierarchies and general information about field KPI and network KPI.
Chapter 3: This chapter firstly introduces which kinds of field KPIs are interested and their definitions, and the second part summarizes field KPI baselines and conditions to achieve the expected KPIs.
Chapter 4: This chapter firstly introduces which kinds of network KPIs are interested and their definitions, and the second part summarizes network KPI baselines and conditions to achieve the expected KPIs.
Chapter 5: This chapter compiles achieved KPIs from global TD-LTE trial networks and early networks.
Chapter 6: This chapter contains a list of useful reference materials.
2 KPI Overview
Key performance indicators (KPIs) are a set of selected indicators used for measuring the network performance which determine the quality of a network.
They are used during RFI/ RFQ phases and during the network/cluster acceptance process. Additionally, network KPIs can be used to monitor a network continuously and warn in time about potential problems.
2.1 KPI Categories
There are several KPIs categories defined, each one looking at a different performance aspect. The names indicate the respective category according to 3GPP 32.410 and 32.450. The names in brackets indicate the terminology commonly used if different from 3GPP.
2.1.1 Accessibility
Accessibility KPIs characterize the availability of a service, or service element to its users. They are expressed as a ratio of successful service requests to the total number of attempts. E.g. Call success ratio.
TD-LTE KPI Optimization Guidelines for RL15TD Version: 1.1
Service request attempts, which are rejected due to insufficient access rights, bad user authentication, or errors in request parameters are not considered and should be taken out of the samples used for calculating the metric.
2.1.2 Retainability (Reliability)
Reliability KPIs can be referred to Service Drop Rates or to Transport Error Rates, that is, the ratio between erroneous or lost data units and the overall number of data units sent. The Service Drop Rates KPIs (i.e. drop rates) can be measured in the field with an FMT (Field Measurement Tool). They represent the ratio between unsuccessful (‘abnormal’) terminations of a service request (e.g. network attach, register, service request) to the overall established services.
For certain services, like bearer creation or VoIP call the service drop rate is also related to the duration of the session. This is motivated by the fact that the possibility of network errors, e.g. hand-over failures is higher if the service duration is longer. In this case, the drop ratio is expressed in [1/s] units instead of [%].
As always-on connectivity is the basic idea for LTE, traditional CDR KPIs may cause abnormal results. CDR per Data Volume has been proposed to solve this problem.
2.1.3 Throughput
Throughput KPIs show the data rates that can be provided to the users of a cell under different radio conditions. The location of the UE in the cell and the number of active UEs in that cell will strongly influence the data rate that can be experienced by one user.
Peak/Average cell throughputs and peak user throughputs are the metrics that characterize the capacity of an LTE network, and the average UL/DL user throughputs along a drive route (i.e. under different radio conditions) represent the end user experience. However, only cell throughput and peak user throughput metrics will be covered in this document.
2.1.4 Latency
Latency KPIs in this document are referred to RTT (Round Trip Time) measurements and to signalling procedures (e.g. setup times)
RTT for data applications is measured with the Ping application of the UEs operating system or with a comparable measurement tool. It records the time difference between sending an ICMP Echo Request to an IP host and the reception of the corresponding ICMP Echo Reply message.
The measurement can be executed between a mobile terminal and an IP host, which is configured to respond to ICMP Echo Requests.
The test is to be done for stationary and mobile end users. The measurement with stationary UE should be performed under average radio link conditions with normal SINR.
As mentioned above, the latency category also refers to the time for signalling procedures on the network or on application level (e.g. RAB service setup, VoIP call setup flow). In case of measuring latencies over the radio link, the time needed to attach the mobile to the network and to setup the necessary radio resources is a significant component.
TD-LTE KPI Optimization Guidelines for RL15TD Version: 1.1
2.1.5 Mobility
Mobility KPIs build a group because all of them are related to hand-over procedures. They could, however, be discussed in the Latency and Accessibility groups as well, because HO-related KPIs are either service interrupt time or success ratio type KPIs.A successful handover is understood to be any successfully performed make before break procedure and it is marked by an entry in the log, which carries the identifier of the new cell (destination).
2.2 Services Levels
In addition to the different categories, KPIs are also defined for different service levels as per 3GPP 36.300. These are:
Application Services
LTE E2E Network Service
Radio Bearer (RB) Service
IP based LTE Bearer Services: X2, S1, S5/S8 and external
Figure 1 shows the LTE Bearer Service Architecture where the KPIs categories above can be mapped into.
Figure 1 LTE Bearer Service Architecture
When it comes to field performance evaluation the most interesting KPIs are the “LTE E2E Network Service” KPIs as they characterize the overall LTE network performance. Another important group is the
TD-LTE KPI Optimization Guidelines for RL15TD Version: 1.1
Application Service KPIs as they show the user’s perceived quality of application services as impacted by the LTE network. The KPIs of this group are service specific. The services of common interest for mobile network operators and therefore, the ones that can be of interest during an acceptance process are:
Web Browsing (HTTP)
Data Upload / Download (FTP)
Email (POP/SMTP)
Conversational Audio Video (VoIP)
The current document concentrates on the Data Upload / Download application service.
2.3 Field and Network KPIs
Field KPIs are the scope of drive testing campaigns either in a trial network or in a live network. They are also called drive test KPIs and can be obtained with a FMT, whereas network KPIs are based on counters collected via NetAct.
Drive test KPIs measurements are recommended in all types of projects (trial, roll-out, swap and optimization). They reflect the network performance along the different drive routes and somehow, reflect the end user experience. They can also be collected in static or indoor locations (e.g. airports, shopping centres, office buildings). However, drive testing campaigns are expensive, time consuming and cannot cover the whole network as the NetAct measurements do. Self-optimization related features in later releases, part of the SON concept, may have an impact on the amount and need of drive testing.
Network KPIs do have statistical significance only if the traffic levels are high enough. This is a reason why, especially during the cluster acceptance process when there is not enough load in the network, the network KPIs may not reflect the network performance and the acceptance process should be based on field KPIs. With the network KPIs it is also unknown if the users causing bad statistical data samples are well inside the coverage area or at the coverage borders. However, network KPIs can be used to monitor networks permanently and highlight potential problems.
Network KPIs are valid and recommended in swap and optimization cases where networks are supposed to be mature and with enough traffic to provide reliable results. In future swap cases (too early at the moment), network KPIs shouldn’t be used to compare with other vendors KPIs as counters and counter triggering conditions are likely to be different.
3 Field KPIs
3.1 Field KPI Definitions
The defined field KPI lists do not present by any means a complete set of KPIs to be used in a customer case. It is the project team/CBT responsibility to define and agree with the customer the final list of KPIs to be used in each and every case.
The measurement procedure of each KPI is just informative, actual test procedure depends on customer’s requirements and may quite different from what has been stated here.
TD-LTE KPI Optimization Guidelines for RL15TD Version: 1.1
The most interested field KPIs are listed below:
Table 1 List of Field KPIs
KPI NameKPI Category
Application Services
PS Data Services (FTP, HTTP etc)
Service Accessibility Ratio [%] Accessibility
Completed Session Ratio [%] Reliability
Single User Throughput [Mbps] Throughput
LTE E2E Network Service
Control Plane
Attach Time [ms] Latency
Attach Success Rate [%] Accessibility
Service Request (EPS) Time [ms] LatencyService Request (EPS) Success Rate [%] Accessibility
Service (EPS) Drop Rate [%] Reliability
Handover Procedure Time [ms] Latency
Handover Success Rate [%] Accessibility
User Plane(LTE) Round Trip Time (RTT) [ms] Latency
(LTE) Single User Throughput [Mbps] Throughput
(LTE) Service Interrupt Time (HO) [ms] Latency
Radio Bearer Services
User Plane(RB) Single User Throughput [Mbps] Throughput
Cell Throughput [Mbps] Throughput
Note that: ‘Single User Throughput’ KPI can be specified on each protocol layer, such as application layer, IP layer, L2 layer (i.e. PDCP/RLC/MAC layer) and L1 layer (i.e. physical layer).
TD-LTE KPI Optimization Guidelines for RL15TD Version: 1.1
3.1.1 (FTP) Service Accessibility Ratio
(FTP) Service Accessibility Ratio [%]
Definition It denotes the probability that the user can establish the necessary bearer (EPS) and access the FTP service successfully.
Service access covers starting the FTP client on the UE, setting up mobile access and creating a TCP connection to the FTP server.
Measurement methodology 1. Set the FMT to generate calls to the FTP Server automatically with the following script/sequence:
a. Connection attempt (network attach)b. Log into the FTP Serverc. FTP Download 800MB file (or other size with transfer duration
>1min)d. Wait 3 sec. after session finishese. FTP Upload 300MB file (or other size with transfer duration
>1min)f. Log out the FTP Server If UE gets into UE deregistered status
between calls, this test also valid to check the Attach Success Rate
g. Connection release (network detach)h. Leave 5 seconds before start next sessioni. Repeat all previous steps (from ‘a’)
2. Continue drive testing till the drive route is completed. 3. Stop the FMT at the end of the route.4. If the route does not generate more than 100 calls repeat the
measurement on the same measurement route.5. The FMT shall record the CRS RSRP, RSRQ and CRS SINR during the
measurement.6. Verify the PS Data Call success rate as per formula below
TD-LTE KPI Optimization Guidelines for RL15TD Version: 1.1
(FTP) Completed Session Ratio [%]
Definition The Completion Session Ratio is the proportion of completed FTP sessions and sessions that were started successfully. The session is not completed if a predefined timer expires, or a network failure occurred that could not be repaired by automatic restarts.
Measurement methodology 1. Set the FMT to generate calls to the FTP Server automatically with the following script/sequence:
a. Connection attempt (network attach)b. Log into the FTP Serverc. FTP Download 800MB file (or other size with transfer duration
>1min)d. Wait 3 sec. after session finishese. FTP Upload 300MB file (or other size with transfer duration
>1min)f. Log out the FTP Server If UE gets into UE deregistered status
between calls, this test also valid to check the Attach Success Rate
g. Connection release (network detach)h. Leave 5 seconds before start next sessioni. Repeat all previous steps (from ‘a’)
2. Continue drive testing till the drive route is completed. 3. Stop the FMT at the end of the route.4. If the route does not generate more than 100 calls repeat the
measurement on the same measurement route.5. The FMT shall record the CRS RSRP, RSRQ and CRS SINR during
the measurement.6. Verify the Completion Session Ratio as per formula below
Assumptions, pre-conditions The FTP transfer shall be executed in binary mode.
FTP Application DL Throughput as measured by the DOS ftp application.
If above measurement methodology is followed the UE needs to attach to the network between FTP UL/DL sessions if the Attach Success Rate wants to be measured.
Formula (logical)FtpSessionSR=
numberof (completed sessions)numberof (successfully started sessions )
∗100 %
Message flow, trigger points Trigger points:
Start: ftp get / put command issued on the UE Stop: after error free execution of the command, ftp returns with a
prompt and indicates the number of bytes transmitted.
Related E2E KPIs (FTP) Service Accessibility Ratio.
TD-LTE KPI Optimization Guidelines for RL15TD Version: 1.1
(FTP) Single User Throughput [Mbps]
Definition After the connection to the FTP server has been successfully established, the parameter describes the average data transfer rate measured over the data transfer phase. I.e. prerequisite to the data transfer is successful Service Access (availability of an EPS bearer and connection to the FTP service). The data transfer phase shall also be successfully terminated.
The throughput is measured on application level in UL/DL direction.
Measurement methodology 1. Set the FMT to generate calls to the FTP Server automatically with the following script/sequence:
a. Connection attempt (network attach)b. Log into the FTP Serverc. FTP Download 800MB file (or other size with transfer duration
>1min)d. Wait 3 sec. after session finishese. FTP Upload 300MB file (or other size with transfer duration
>1min)f. Log out the FTP Server If UE gets into UE deregistered status
between calls, this test also valid to check the Attach Success Rate
g. Connection release (network detach)h. Leave 5 seconds before start next sessioni. Repeat all previous steps (from ‘a’)
2. Continue drive testing till the drive route is completed. 3. Stop the FMT at the end of the route.4. If the route does not generate more than 100 calls repeat the
measurement on the same measurement route).5. The FMT shall record the CRS RSRP, RSRQ and CRS SINR during
the measurement.6. Verify the average DL/UL throughput of each session and determine
the final average throughput of all sessions.7. The measured mean throughputs can be displayed in diagrams as
function of time to show the impact of FTP slow start.
Assumptions, pre-conditions The initial status of UE before starting the application: EMM-REGISTERED, ECM-CONNECTED. If above measurement methodology is followed the UE needs to attach to the network between FTP UL/DL sessions if the Attach Success Rate wants to be measured.
The FTP application shall be executed in binary mode, and the file to be transferred shall be binary.
Formula (logical)FtpMeanTp=
transferred¿ [bytes ]∗8transfer time [s ]
∗10−6
Message flow, trigger points The average throughput is measured from opening the data connection to the end of the successful transfer of the content (file, e-mail or web page).
Trigger points for ftp upload / download are defined in [ETSI102.250-2]
TD-LTE KPI Optimization Guidelines for RL15TD Version: 1.1
3.1.4 Attach Time
Attach Time (Control Plane) [ms]
Definition With Attach, the mobile terminal registers at the LTE network. At the end of the procedure the UE is authenticated, and a default (non-GBR) bearer is established.
The Attach Time is the interval between the RRC CONNECTION REQUEST (carrying the Attach Request) and the reception of a positive response by the UE (Attach Complete).
Measurement methodology 1 LTE UE/terminal with FMT Set FMT to generate data calls to the data server automatically Check the UE DEGERISTERED between data calls Start drive test and continue driving until drive route is completed
and enough samples are collected (re-drive route if necessary) Verify the Attach Time
Assumptions, pre-conditions UE status before measurement: EMM-DEREGISTERED
UE locked to LTE network.
Formula (logical) AttachTime [ms ]=tAttachComplete−tAttach Request
Message flow, trigger points Trigger points on UE R interface (see Figure 2, and refer to [3GPP23.401]):
Begin: 1. RRC CONNECTION REQUEST sent by UE carrying L3 NAS: Attach Request in its body.
End: 21. RRC DIRECT TRANSFER message with L3 NAS: Attach Complete in its body sent by UE (refer to [3GPP24.008]).
Related E2E KPIs a) Attach Success Rate.b) No 3GPP defined KPI for Attach Time found exists, or
measurements defined in R8 documents.
Detailed attach procedure with initial EPS bearer establishment is shown below.
For initial Attach, the following specifics of the message flow in Figure 2 have to be considered: A signaling bearer (SRB1) is to be established, before an Attach Request can be sent, i.e. the UE is
expected execute a contention based Random Access (RA) procedure. Step 3. is not executed, since the IMSI is queried from the UE directly using Step 4. The optional
Identity Request / Response messages in step 5.b do not have to be exchanged if the MEI is already returned by the UE using the Security Mode Complete message as part of step 5.a.
Steps 7-11 are not executed, no EPS bearers are to be deleted.
TD-LTE KPI Optimization Guidelines for RL15TD Version: 1.1
3.1.5 Attach Success Rate
Attach Success Rate (Control Plane) [%]
Definition The Attach Success Rate is defined as the ratio between the number of successful registrations (RRC DIRECT TRANSFER: Attach Complete) and the number of all requests (RRC CONNECTION REQUEST: Attach Request). It is the probability that a user can attach to the LTE network at any moment of time.
The calculated success ratio figure excludes attach requests, which are rejected by authentication failures. On the other hand, network attach requests which are terminated by timer expiry (due to the unavailability of some LTE resource) are considered as unsuccessful registrations.
If the success rate is calculated on the eNB by counting incoming RRC requests, RRC CONNECTION REQUEST retries are to be excluded, since they would increase the overall number of establishment attempts, and thus reduce the success ratio.
Measurement methodology
For defining the success ratio, the same series of “Attach Time” measurements can be used:
1 LTE UE/terminal with FMT Set FMT to generate data calls to the data server automatically Start drive test and continue driving until drive route is completed and enough
samples are collected (re-drive route if necessary) Verify the Attach Success Rate as per formula below
The KPI cannot be measured as end-user perceived ratio on certain mobiles, which do not indicate network attachment status to the end-user.
TD-LTE KPI Optimization Guidelines for RL15TD Version: 1.1
3.1.6 Service Request (EPS) Time, UE Initiated
Service Request Time (EPS Bearer Setup), UE Initiated, Control Plane [ms]
Definition Time taken by the LTE network to setup an EPS bearer on request by the UE. The EPS bearer can be new (dedicated) or an existing one (e.g. the default EPS bearer). The latter is needed to re-assign Uu radio and S1 bearer resources to the existing EPS bearer of a previously idle UE.
The EPS bearer has to be created before IP packets can be exchanged.
The UE Initiated EPS Bearer Setup Time is the interval between the submission of the message RRC CONNECTION REQUEST: Service Request (refer to [3GPP36.331]) and the reception of the RRC CONNECTION RECONFIGURATION response on the UE. The Bearer Setup procedure is based on the Service Request procedure according to [3GPP23.401] and [3GPP36.300].
Measurement methodology
1 LTE UE/terminal with FMT Set FMT to generate data calls to the data server automatically Start drive test and continue driving until drive route is completed and
enough samples are collected (re-drive route if necessary) Verify the EPS Bearer Setup Time Note below that UE pre-status is EMM-REGISTERED so this KPI excludes
the Attach time
Assumptions, pre-conditions
UE status before measurement: EMM-REGISTERED and idle initial state.
UE locked to LTE network.
Formula (logical) Service Request Time [ms ]=t RRCConnectionReconfigurationComplete−tRRCConnectionRequest
Message flow, trigger points
The message flow diagram is shown in Figure 3 and Figure 4. The user initiated EPS bearer setup time is measured between the trigger points BUE and EUE on the UE R interface:
Begin: RRC CONNECTION REQUEST(NAS: Service Request) sent by UE End: corresponding RRC CONNECTION RECONFIGURATION
COMPLETE message or first PDU in UL sent by the UE (refer to [3GPP24.008]).
Related E2E KPIs a) Attach Time.b) No 3GPP defined KPI for EPS bearer setup found (exists).
Detailed EPS bearer setup procedure is shown below:
TD-LTE KPI Optimization Guidelines for RL15TD Version: 1.1
UE eNodeB MME S-GW / P-GW HSS
LTE-Uu S1- MME S11 S6a
S1-AP: INITIAL CONTEXT
SETUP REQUEST
PM Counter: EPS_SETUP_ATT
RRC SECURITY MODE COMMAND
RRC CONNECTION RECONFIGURATION
COMPLETE
S1-AP: INITIAL CONTEXT
SETUP COMPLETE
L2 SCTP: Update Bearer Request
L2 SCTP: Update Bearer Response
First UL PDU
T1
T2
RRC CONNECTION RECONFIGURATION
RRC SECURITY MODE COMMAND COMPLETE
Setup user plane RB
Means „Delay DL Packet Notification Request“ to stop/enable DL data transferPM Counter:
EPS_SETUP_SUCC
End measurement
First DL PDU
EUE
EN
S1-AP: INITIAL CONTEXT
SETUP FAILURE
Figure 4 EPS Bearer Setup Procedure, Part II
The service request times for UE initiated and for network initiated EPS bearer setups are measured between the trigger points BUE and EUE, and between BN and EN, respectively. These trigger points are colored green in the diagram.
TD-LTE KPI Optimization Guidelines for RL15TD Version: 1.1
3.1.7 Service Request (EPS) Success Rate
Service Request (EPS) Success Rate, Control Plane [%]
Definition Ratio between successfully established EPS bearers compared to the overall number of EPS bearer establishment attempts. It corresponds to the probability that a user or the LTE network can establish an EPS bearer at any moment in time.
Requests which are terminated by timer expiry (due to the unavailability of some LTE resource) are considered as unsuccessful attempts. Authentication errors (requests rejected by the MME) are included in the total number of failures.
Only the first RRC CONNECTION REQUEST is to be considered, since counting retries of the same message would increase the overall number of bearer establishment attempts, and thus reduce the success ratio.
Measurement methodology
1 LTE UE/terminal with FMT Set FMT to generate data calls to the data server automatically Start drive test and continue driving until drive route is completed and
enough samples are collected (re-drive route if necessary) Verify the EPS Bearer Setup Success Rate
Note below that UE pre-status is EMM-REGISTERED so this KPI excludes the Attach time
Assumptions, pre-conditions
UE status before measurement: EMM-REGISTERED and idle initial state.
TD-LTE KPI Optimization Guidelines for RL15TD Version: 1.1
3.1.8 Service (EPS Bearer) Drop Rate
Service (EPS Bearer) Drop Rate, Control Plane [%]
Definition
Ratio between abnormally released bearers, and the overall number of established EPS bearers. An abnormal release is defined as any EPS bearer termination, which was not triggered by the mobile user (from UE side). Thus, it reflects the probability that an established bearer gets aborted due to insufficient network resources.
Dropping the bearer gets visible to the end-user if an application service is actively using it. If the application automatically re-establishes the bearer, it remains unnoticed by the user.
Measurement methodology
1 LTE UE/terminal with FMT Set FMT to generate data calls to the data server automatically Start drive test and continue driving until drive route is completed and
enough samples are collected (re-drive route if necessary) Verify the EPS Bearer Drop Rate
Note below that UE pre-status is EMM-REGISTERED so this KPI excludes the Attach time.
Assumptions, pre-conditions
UE status before measurement: EMM-REGISTERED and idle initial state.
UE locked to LTE network.
Formula (logical) EPSBearerDR=numberof (dropped calls)numberof (successful calls )
∗100 %
Message flow, trigger points
Trigger points on UE R interface:
Success : submission of the message RRC CONNECTION RECONFIGURATION COMPLETE by the UE, refer to [3GPP36.331]
Drop : successful establishments minus terminations by the user, i.e. the UE submitting a L3 DETACH REQUEST (refer to [3GPP24.008]) carried within the RRC message UL NAS TRANSPORT (refer to [3GPP36.331]).
For the calculation of a network initiated EPS bearer drop rate, EPC initiated L3 DETACH REQUEST-s are to be considered.
Related E2E KPIs Service Request (EPS) Success Rate.
TD-LTE KPI Optimization Guidelines for RL15TD Version: 1.1
3.1.9 Handover Procedure Time
Handover Procedure Time, Control Plane [ms]
Definition It denotes the total time needed for the handover procedure as seen by the UE. It begins by receiving a Handover Command from the source eNB and ends by sending the Handover Confirm response to the target eNB by the UE. The value of the KPI also depends of the handover scenario. The following HO scenarios are distinguished (though not directly seen by the UE):
intra- and inter-frequency HO inter eNB HO via X2/S1 interface
Measurement methodology
1 LTE UE/terminal with FMT Set FMT to generate data calls to the data server automatically Start drive test and continue driving until drive route is completed and
enough samples are collected (re-drive route if necessary) Verify HO time for the different HOs
The handover time shall be measured on the UE R interface.
Assumptions, pre-conditions
Since Inter-RAT HO is not part of RL15 the only HO scenarios considered are intra LTE. Therefore, the LTE UE must be locked to LTE network.
UE status before measurement: registered and connected.
Formula (logical) Average value from all measured samples.
HOProcedureTime [ms ]=tHOConfirm−t HOCommand
Message flow, trigger points
The HO time is seen by the UE as the elapsed time between:
Start: UE receives Handover Command in the body of an RRC Connection Reconfiguration request from source eNB.
Stop: UE sends Handover Confirm response to target eNB in an RRC Connection Reconfiguration Complete message.
Related E2E KPIs a) (LTE) Service Interrupt Time (HO), Handover Success Rateb) No 3GPP defined KPI for HO procedure time found.
TD-LTE KPI Optimization Guidelines for RL15TD Version: 1.1
3.1.10 Handover Success Rate
Handover Success Rate, Control Plane [%]
Definition The Handover Success Rate is the ratio between successfully executed (committed) HO procedures and the number of all Handover attempts.
Measurement methodology
1 LTE UE/terminal with FMT Set FMT to generate data calls to the data server automatically Start drive test and continue driving until drive route is completed and
enough samples are collected (re-drive route if necessary) Verify the Handover Success Rate
The measurement is to be executed for different HO scenarios.
Total # : Handover Requests sent by Source eNB to Target eNB (HO via X2), or sent by MME to Target eNB(HO via S1).
Success # : UE Context Release message received by Source eNB from Target eNB(HO via X2), or UE Context Release Command received by Source eNB from MME(HO via S1).
In case of intra eNB HO procedures, no Handover Request is sent. Both trigger points are counted by the Source eNB internally.
TD-LTE KPI Optimization Guidelines for RL15TD Version: 1.1
3.1.11(LTE) Round Trip Time
Round Trip Time (RTT), User Plane [ms]
Definition RTT in UL is the interval between sending a datagram by the UE & receiving the corresponding reply from an IP peer entity connected to the Gi interface of the P-GW.
RTT in DL is the interval between sending a datagram to the UE & receiving the corresponding reply by the IP host (peer entity).
Measurement methodology
Stationary users in different cell positions, which are uniformly distributed across the cell.
RTT is measured with the Ping application between the UE and an IP host (peer entity). As average RTT figure, the output of the Ping application is used (e.g. “MS-DOS >ping -n count -l size host name”). Size of ICMP packets: 32, …, MTU bytes.
Ping shall be executed at least 100 times to reduce the impact of the first ICMP message, which triggers the setup of radio bearer establishments in UL and DL directions.
Record the average RTT shown on the screen after the application has run.
Repeat the test 5 times in each location under different SINR.
Note: Ping tests can also be carried out as part of a drive test sequence in which case, it is not possible to guarantee certain SINR levels
Assumptions, pre-conditions
UE status before measurement: registered and connected. An IP address is assigned to the UE.
The IP peer entity shall be located as close as possible to the SGi interface. Usually, 1 hop away the SAE-GW. Additional delays between the P-GW and the IP host shall be avoided or minimized.
Formula (logical) RTT [ms ]=t ICMPEcho Reply−t ICMP EchoRequest
Message flow, trigger points
Trigger points both from UE, and from IP peer entity view:
Start: ICMP ECHO REQUEST Stop: ICMP ECHO REPLY
Related E2E KPIs No 3GPP defined KPI for LTE RTT found.
3.1.12 (LTE) Single User Throughput
(LTE) Single User Throughput, User Plane [Mbps]
Definition The metric describes the data speed available to one user of the LTE
TD-LTE KPI Optimization Guidelines for RL15TD Version: 1.1
network on UDP/IP level. It is given as the maximum (95%-ile) value that can be observed over a short period of time (e.g. of 1s) and as a mean value that characterizes longer data transfer periods (minutes). Its value distribution over the radio cell is given as a function of the SINR. The maximum value is often referred to in the literature as instantaneous "Peak Throughput" that is achieved in optimal radio conditions. The
user throughput can be given for a single user active in the cell (single user data rate), or to one of several concurrently active users.
Measurement methodology
Stationary and mobile user covering all kinds of cell positions: near, medium, far, including different HO scenarios. The data rate is measured in UL/DL direction with UDP/IP traffic over a time period of several minutes.
Assumptions, pre-conditions
The IP host used as traffic generator (e.g. with Iperf) should be connected directly to the SGi interface. Additional delays and bandwidth restrictions between the S/P-GW and the server need to be excluded.
Formula (logical)UserTp [Mbps ]=
transferred¿ [bytes]∗8transfertime [ s]
∗10−6
Message flow, trigger points
Trigger points on the UE R interface:
• UL: UE sending UDP/IP packets to the IP peer entity.
• DL: IP peer entity sending UDP/IP packets to the UE.
Related E2E KPIs Cell Throughput, (FTP) Single User Throughput, (RB) Single User Throughput.
The Single User Throughput KPI has been defined on E2E network level to verify if the capacity of large non-GBR EPS bearers (QoS profile with large maximum bit rates) can in fact be exploited by IP based services. The same holds for the E-RAB and RB bearers being part of the EPS. Especially, the maximum achievable throughput of the RB bearer is interesting, since it is the critical resource of the LTE network. For this reason, the Peak Throughput is defined as a separate KPI on the RB level (see KPI: (RB) Single User Throughput), too. Because of this relationship (containment) between EPS, E-RAB and RB, user throughput KPIs can be verified with the same series of measurements. The KPIs are interesting as values calculated over the full duration of long data transfers (~minutes). Of equal interest are peak values observed for a short time period only (~seconds). Please note that throughput KPIs have also been defined for application services, e.g. Single User Throughput KPI of the Data Download / Upload service, but these do not characterize the capability of the LTE network alone, but also depend on the application service and its protocols.
3.1.13 (LTE) Service Interrupt Time (HO)
(LTE) Service Interrupt Time (HO), User Plane [ms]
Definition Service interrupt is the discontinuity (increased packet inter-arrival time) of an
TD-LTE KPI Optimization Guidelines for RL15TD Version: 1.1
IP packet flow caused by hand-over. It is the interval between the last sent/received user IP packet of a continuous UL/DL data stream in the old cell, and the first sent/received user IP packet in the new cell (user plane break).
The value of the KPI also depends of the handover scenario.
Measurement methodology
1 LTE UE/terminal with FMT
Set FMT to generate data calls to the data server automatically
Start drive test and continue driving until drive route is completed and enough samples are collected (re-drive route if necessary)
Verify the Service Interrupt Time
The measurement is to be executed for different HO scenarios: intra eNB, inter eNB via X2/S1 etc.
Only mobile user scenarios are considered. FTP upload/download data transfer, or real-time media flow during the handover.
Assumptions, pre-conditions
It is assumed that the X2 interface is enabled between the source and target eNBs (for inter eNB HO scenarios).
LTE terminal must be locked into the LTE network to measure the Service Interrupt Time.
Formula (logical) Service Interrupt Time [ms ]=tFirstPacket ¿ /¿TeNB¿−tLast Packet ¿/¿SeNB ¿
Message flow, trigger points
The trigger points for service interrupt time (from UE point of view):
Start: UE sends/receives last packet to/from source eNB
Stop: UE sends/receives first packet to/from target eNB.
Related E2E KPIs Handover procedure time.
3.1.14 (RB) Single User Throughput
(RB) Single User Throughput, User Plane [Mbps]
Definition The metric describes the RB throughput achievable by one user. It can be given as single user throughput if only one user is active in the cell, or as multi-user throughput for a given number of concurrently active users. The user throughput distribution over the cell is given as function (CDF) of the radio conditions characterized by the SINR.
The time variation of the KPI value in a given cell position is given with its maximum (95%-ile) (also referred to as instantaneous “Peak User Throughput”), and with its average (mean) value.
TD-LTE KPI Optimization Guidelines for RL15TD Version: 1.1
Measurement methodology
Stationary and mobile user covering all kinds of cell positions: near, medium, far, including different HO scenarios.
The eNB assigns all resources (resource blocks) to this user, except the resource blocks used for signaling and controlling. With the help of trace analysis, PDCP, RLC, MAC and PHY layer throughputs can be defined.
With and w/o concurrency in UL / DL traffic.
The mean user data rate is calculated over a few minutes, peak data rate values are averaged over short periods of time (e.g. 1s).
Assumptions, pre-conditions
All transport bearers between the UE and IP peer entity should have higher capacity than the radio link in order to avoid bandwidth bottlenecks.
Formula (logical)RBUserTp [Mbps ]=
transferred¿[bytes]∗8transfer time [s ]
∗10−6
Message flow, trigger points
Trigger events on the eNB Uu interface or the UE R interface:
UL: eNB receiving PDUs, or UE sending PDUs DL: eNB sending PDUs, or UE receiving PDUs
Measured at the upper SAPs of the protocols PDCP, RLC, MAC or PHY.
Related E2E KPIs (LTE) Single User Throughput, Cell Throughput.
3.1.15 Cell Throughput
(RB) Single User Throughput, User Plane [Mbps]
Definition The metric shows the sustainable aggregate throughput of the cell (in UL/DL) available to “n” stationary users distributed uniformly in the cell and running a typical mix of applications. The “cell throughput” is the sum of all bits transported in all resource blocks carrying PDUs (i.e. bits in ULSCH/DL-SCH transport blocks) during one second.
The cell capacity is also given as peak value (called peak cell capacity, or throughput), which is defined as the aggregate throughput of “n” users all located in best radio conditions.
The cell throughput value is defined here on PHY level, but could be given for other protocol levels (UDP/IP, PDCP, RLC, MAC), too. When the (peak, average) cell throughput is expressed on UDP/IP level, it corresponds to the (peak, mean) user throughput value at comparable radio conditions.
Measurement methodology
Stationary users uniformly distributed in the cell. UE categories according to application mix. Peak value measured with one user in LoS cell position using UDP/IP load to approach the full buffer condition.
Measured with UDP/IP traffic on PHY layer (UL-SCH/DL-SCH transport block
TD-LTE KPI Optimization Guidelines for RL15TD Version: 1.1
bits) over several minutes. With the help of trace analysis PDCP, RLC and MAC layer values can also be given.
Assumptions, pre-conditions
All IP transport bearers (S1, S5/S8, SGi) between the UE and IP peer entity should have higher bandwidth than the cell capacity to avoid bandwidth bottlenecks.
Formula (logical)CellTp [Mbps ]=
transferred¿ [bytes ]∗8transfer time [s ]
∗10−6
Message flow, trigger points
Trigger events on the eNB Uu interface or the UE R interface:
UL: eNB receiving PDUs, or UE sending PDUs DL: eNB sending PDUs, or UE receiving PDUs
Related E2E KPIs (RB) Single User Throughput.
3.2 Test Conditions and Target Values
Field KPI Target Values CommentsThroughput
Single User UL Peak
Throughput
In case UL/DL conf=2:2(#1) and special subframe conf=10:2:2(#7): 15.4M
In case UL/DL conf=3:1(#2) and special subframe conf=10:2:2(#7): 7.5M
CMCC Hangzhou TD-LTE trial:*Cat 3 UE is considered and MCS20 is the max MCS in UL for RL15.*PUCCH overhead is 4RB and PDCCH overhead is 3 OFDM symbols.*L3 TCP throughput is considered.*Network is unloaded.*System bandwidth is 20M.
Single User DL Peak
Throughput
In case UL/DL conf=2:2(#1) and special subframe conf=10:2:2(#7):TM3: 59.6MTM7: 34M
In case UL/DL conf=3:1(#2) and special subframe conf=10:2:2(#7):TM3: 80.7MTM7: 40.3M
Cell UL Average
Throughput
In case UL/DL conf=2:2(#1) and special subframe conf=10:2:2(#7):0% load: 14.5M50% load: 13.3M70% load: 15.6M
In case UL/DL conf=3:1(#2) and special subframe conf=10:2:2(#7):0% load: 7.6M50% load: 6.5M
CMCC Hangzhou TD-LTE trial:*Cat 3 UE is considered and MCS20 is the max MCS in UL for RL15.*PUCCH overhead is 4RB and PDCCH overhead is 3 OFDM symbols.*L3 TCP throughput is considered.*System bandwidth is 20M.*20 UEs distributed by 2:4:8:6.*DL MIMO is TM3/TM7 adaptive and UL is SIMO.*eNB SW version is different for each neighbor load: RL15_2_CD1.0 for 0% load, RL15_2_CD2.0 for 50% load and RL15_2_CD3.0 for 70% load.
Cell DL Average
Throughput
In case UL/DL conf=2:2(#1) and special subframe conf=10:2:2(#7):0% load: 35.6M50% load: 21.77M70% load: 24.82M
In case UL/DL conf=3:1(#2) and special subframe conf=10:2:2(#7):0% load: 40M
In case UL/DL conf=2:2(#1) and special subframe conf=10:2:2(#7):0% load: 32ms
CMCC Hangzhou TD-LTE trial:*Drive route includes both intra-eNB handover and inter-eNB handover (via X2)..*C-plane latency is counted as MAC layer latency, and U-plane latency is counted as PDCP layer latency, which is not strictly complied with CMCC's test specification.
Handover Latency
(User Plane, intra-eNB or inter-eNB via
X2)
In case UL/DL conf=2:2(#1) and special subframe conf=10:2:2(#7):0% load: 61ms
Accessibility
Attach Success Rate
In case UL/DL conf=2:2(#1) and special subframe conf=10:2:2(#7):0% load: 100%50% load: 99.17%
CMCC Hangzhou TD-LTE trial:*2.6G frequency band with 20M system bandwidth.*Drive test, speed is about 30Km/h.*Attach SR is defined as: #(Attach complete) / #(UE Power on).*RRC Connection Establishment SR is defined as: #(RRCConnectionReconfigurationComplete) / #(Data service request)*Handover SR is defined as:#(RRCConnectionReconfigurationComplete containing Handover Confirm to Target eNB) / #(Handover Request)*DR is defined as:#(RRCConnectionRelease and/or without application-layer throughput for more than 10s) / #(RRCConnectionReconfigurationComplete)
RRC Connection
Establishment Success Rate
In case UL/DL conf=2:2(#1) and special subframe conf=10:2:2(#7):0% load: 99%50% load: 97.84%
Handover Success Rate
In case UL/DL conf=2:2(#1) and special subframe conf=10:2:2(#7):0% load: 100%50% load: 98.37%
Reliability Drop Rate
In case UL/DL conf=2:2(#1) and special subframe conf=10:2:2(#7):0% load: 2%50% load: 3.42%
TD-LTE KPI Optimization Guidelines for RL15TD Version: 1.1
4 Network KPIs
4.1 Network KPI Definitions
The defined network KPI lists do not present by any means a complete set of KPIs to be used in a customer case. It is the project team/CBT responsibility to define and agree with the customer the final list of KPIs to be used in each and every case.
The most interested network KPIs are listed below. Formula is indicated with the counter ID (e.g. M8007C1) which is unique for each counter instead of the counter name. More information about these KPIs and other network KPIs can be found in RISE database (refer to RL20 instead).
TD-LTE KPI Optimization Guidelines for RL15TD Version: 1.1
4.2 Test Conditions and Target Values
TBD
5 Appendix I: Trial Network KPIs
5.1 CMCC Large Scale Trial
5.1.1 Test Environment and Baseline Configuration
Test environment of CMCC large scale trial in Hangzhou is illustrated as below. There are 24 eNBs and 50 cells in total. Selected route for DT is also shown.
Figure 5 Test Environment of CMCC Large Scale Trial in Hangzhou
Baseline configuration is shown as below.
Table 3 Baseline Configuration of CMCC Large Scale Trial in Hangzhou
TD-LTE KPI Optimization Guidelines for RL15TD Version: 1.1
Carrier Frequency 2.6G
System Bandwidth 20M
Frame Structure
Normal CPUL/DL Configuration #1 (2:2) Special Subframe Configuration #7 (10:2:2)
CFI 3
Transmission ModeDL: TM3/TM7 AdaptiveUL: SIMO
Power ControlEnable PUCCH, PUSCH, and Sounding RS Power Control
HARQ Enable
AMC EnableeNB Transmission Power 8×5W
Random Access Contention-based
UE location is selected based on CRS SINR according to the following rules:
*Very Good: >22dB
*Good: 15dB~20dB
*Medium: 5dB~10dB
*Poor: -5dB~0dB
5.1.2 Field Results
Note that interference/load is generated by real UEs distributed evenly in medium point and poor point, performing UL/DL FTP traffic at the same time, which results in 50%/70%/100% load in DL and 5dB/8dB/11dB IoT in UL respectively.
Table 4 Field Results of CMCC Large Scale Trial in Hangzhou
KPI Name Test Condition Field Results Comments
Attach SR Drive test(~30km/h), without load or under 50% load.
Without load: 100%50% load: 99.17%
Defined as:Count(Attach complete)/Count(UE Power on)
RRC Connection Establishment SR
Drive test(~30km/h), without load or under 50% load. preambleTransMax=3.
Without load: 99%50% load: 97.84%
Defined as:Count(RRCConnectionReconfigurationComplete)/Count(data service request)
TD-LTE KPI Optimization Guidelines for RL15TD Version: 1.1
Poor 51ms 24ms 29ms
1500 Bytes
Very Good 41ms 26ms 31ms
Good 41ms 18ms 30msMedium 51ms 19ms 30ms
Poor 61ms 23ms 33ms
Multiple users ping latency, with 50% load
PayloadTest
Condition
Max Latency
Min Latency
Avg Latency
32 Bytes
Good 43 24 28
Medium 49 18 29
Poor 43 24 28
1500 Bytes
Good 57 26 37
Medium 69 24 38
Poor 66 31 39
*Control-plane Latency, also idle-to-active latency
Control-plane latency is defined as the duration between MSG1 of random access and the RRCConnectionConfigurationComplete sent by UE to eNB.
With 50% load:
Test Conditio
n
Min Latency
Max Latency
Avg Latency
Very Good
100 110 105
Good 100 110 107Medium 100 120 107
Poor 100 130 107
*Handover Latency
Handover latency of control-plane is defined as the duration between MeasurementReport sent by UE to source eNB and the RRCConnectionConfigurationComplete sent by UE to target eNB.
Handover latency of user-plane is defined as the duration between UE/source eNB receives the last packet from source eNB/UE and UE/target eNB receives the first packet from target eNB/UE in DL and UL respectively.
TD-LTE KPI Optimization Guidelines for RL15TD Version: 1.1
UE location is selected based on following radio conditions:
-Near: CRS SINR >25dB
-Middle: 12dB<CRS SINR <15dB
-Far: 2dB<CRS SINR<5dB
Field results are:
Note that, herein definition of idle-to-active latency, which is the duration between ServiceRequest and RRCConnectionSetupComplete, is not correct.
*RTT, also ping latency
Field results(pre-scheduling is on) are:
*Single User Throughput
Field results are:
In above figure, “Config-1” means UL/DL configuration 1 with special subframe configuration 7, and “Config-2” means UL/DL configuration 2 with special subframe configuration 7.
