SaT5G (761413) D4.3 May 2020

D4.3

Multi-link and Heterogeneous Transport - Analysis, Design and Proof of Concepts

Satellite and Terrestrial

Network for 5G

SaT5G (761413) D4.3 May 2020

Topic ICT-07-2017

Project Title Satellite and Terrestrial Network for 5G

Project Number 761413

Project Acronym SaT5G

Contractual Delivery Date Aug 2019 (M27)

Actual Delivery Date 30/12/2019

Contributing WP WP4.3

Project Start Date 01/06/2017

Project Duration 33 months

Dissemination Level PU

Editor OA

Contributors SES, ADS, TNO, i2CAT

Document History

Version Date Modifications Source

0.1 28/06/18 Initial structure OA

0.8 17/10/18 Multi-Access state of the art and Multi-Access at 3GPP OA

0.85 1/11/19 QoS in Satcom TNO

0.9 10/11/19 MPTCP and MPQUIC BT

1.0 18/12/19 Prototypes, PEP and Standardization OA

SaT5G (761413) D4.3 May 2020

Name Organization

Mamoutou Diarra OA

Relja Djapic TNO

M. (Miodrag) Djurica TNO

Philip Eardley BT

Thierry Masson OA

Franck Messaoudi OA

Luc Ottavj OA

Boris Tiomela Jou ADS

Simon Watts AVA

Hamzeh Khalili I2CAT

Ning Wang UoS

SaT5G (761413) D4.3 May 2020

Table of Contents

Table of Contents .................................................................................................................................. 4

List of Figures ....................................................................................................................................... 6

List of Tables ......................................................................................................................................... 7

List of Acronyms ................................................................................................................................... 8

Executive Summary ............................................................................................................................ 10

1 Introduction .................................................................................................................................. 11

2 Multi-access State-of-the-Art ....................................................................................................... 13

2.1 PEP ..................................................................................................................................... 13

2.2 Type of flows ....................................................................................................................... 15

2.3 Path Selection algorithms .................................................................................................... 15

2.4 Multipath Methods and protocols ........................................................................................ 16

2.4.1 Layer 3 ................................................................................................................................ 16

2.4.2 MPTCP ................................................................................................................................ 18

2.4.2.1 MPTCP Hybrid solution & proxy ...................................................................................... 20

2.4.2.2 Non-TCP traffic................................................................................................................ 20

2.4.2.3 Practical work on MPTCP and satellite systems ............................................................. 21

2.4.3 MPQUIC .............................................................................................................................. 22

2.4.3.1 QUIC ............................................................................................................................... 22

2.4.3.2 QUIC over satellite links .................................................................................................. 24

2.4.3.3 Multipath QUIC ................................................................................................................ 26

3 3GPP and Multi-Access ............................................................................................................... 27

3.1 AT3S Function .................................................................................................................... 27

3.2 AT3S architecture framework .............................................................................................. 27

3.2.1 Multi-access Sessions ......................................................................................................... 29

3.2.2 User Plane AT3S methods (AT3S Functions) ..................................................................... 30

3.2.3 User plane MPTCP method (MPTCP function) ................................................................... 31

3.2.4 AT3S limitations .................................................................................................................. 31

3.3 Backhaul Traffic Steering, Switching, and Splitting: BT3S .................................................. 31

3.3.1 5G UE using an AT3S or MPTCP function .......................................................................... 32

3.3.2 UE not using an AT3S or MPTCP function .......................................................................... 33

3.3.3 Regular hosts connected to the hybrid backhaul gateway .................................................. 33

3.4 Mapping AT3S to SaT5G architecture options .................................................................... 34

4 Satcom QOS in to 5G .................................................................................................................. 36

4.1 Connection setup ................................................................................................................ 36

4.2 QoS in 5G ........................................................................................................................... 36

4.2.1 QoS Profiles ........................................................................................................................ 37

4.2.2 QoS Rules ........................................................................................................................... 39

4.2.3 Legacy Satellite QoS (Class of Service) .............................................................................. 40

4.3 QoS in SaT5G hybrid backhaul ........................................................................................... 42

SaT5G (761413) D4.3 May 2020

5 5G multi-link prototypes ............................................................................................................... 44

5.1 MPTCP multi-link prototype ................................................................................................. 44

5.1.1 Setup ................................................................................................................................... 45

5.1.2 Results ................................................................................................................................ 47

5.2 MPQUIC multi-link prototype ............................................................................................... 48

5.2.1 Setup ................................................................................................................................... 48

5.2.2 Software .............................................................................................................................. 49

5.2.3 Test Method ........................................................................................................................ 49

5.2.4 Test Results ........................................................................................................................ 50

5.2.4.1 Sequential mode ............................................................................................................. 51

5.2.4.2 Parallel mode .................................................................................................................. 52

6 Standardization ............................................................................................................................ 54

7 Conclusion ................................................................................................................................... 55

Bibliography ........................................................................................................................................ 56

SaT5G (761413) D4.3 May 2020

List of Figures

Figure 1-1 Inter-relationships with other works in SaT5G ................................................................... 12 Figure 2-1 TCP PEP in action ............................................................................................................. 14 Figure 2-2 Hybrid access using multiple paths between gateways [41] .............................................. 19 Figure 2-3 Comparison of TCP and QUIC protocol stacks.................................................................. 23 Figure 2-4 Structure of QUIC packet with long-header for IETF QUIC (version 14)............................ 24 Figure 2-5 Satellite system using split TCP with performance enhancing proxies .............................. 25 Figure 3-1 Non roaming AT3S architecture - initial high level view [8] ................................................ 28 Figure 3-2 MA PDU session................................................................................................................ 29 Figure 3-3 Interaction between AT3SF's of UE, SMF, PCF & UPF ..................................................... 30 Figure 3-4 AT3S using WLAN and 5G interfaces ................................................................................ 31 Figure 3-5 Multi-link in backhaul using BT3S ...................................................................................... 32 Figure 3-6 BT3S with AT3SF UE ........................................................................................................ 33 Figure 3-7 BT3S with non AT3SF UE ................................................................................................. 33 Figure 3-8 BT3S with regular (non 5G) hosts...................................................................................... 34 Figure 3-9 Backhaul implementation options identified by SaT5G ...................................................... 34 Figure 4-1 Classification principle, User Plane QoS Flows marking and AN Resources mapping ...... 40 Figure 4-2 Adaptation of QoS configurations between satellite and terrestrial networks .................... 42 Figure 5-1 GTP header detection ........................................................................................................ 44 Figure 5-2 Packet layout over the backhaul links ................................................................................ 45 Figure 5-3 Bandwidth aggregation and latency reduction ................................................................... 45 Figure 5-4 MPTCP prototype testbed ................................................................................................. 46 Figure 5-5 IUG virtual and physical interfaces .................................................................................... 46 Figure 5-6 ING virtual and physical interfaces .................................................................................... 46 Figure 5-7 MPQUIC testbed................................................................................................................ 49 Figure 6-1 Multi-link backhaul over two links as introduced in 3GPP .................................................. 54

SaT5G (761413) D4.3 May 2020

List of Tables

Table 1 Standardized 5QI to QoS characteristics mapping [67].......................................................... 37 Table 2 An example of mapping between satellite CoS and 5QI ........................................................ 41 Table 3 MPTCP expected performance .............................................................................................. 48 Table 4 PSBOL+ Offload experimental results ................................................................................... 48 Table 5 MPQUIC experimental results .................................................................................................... Table 6 Sequential Mode experimental results ....................................................................................... Table 7 Parallel Mode experimental results ............................................................................................

SaT5G (761413) D4.3 May 2020

List of Acronyms

Acronym Description

3GPP 3rd Generation Partnership Project

4G 4th Generation mobile network

5G 5th Generation mobile network

5QI 5G QoS Identifier

AF Application Function

AMF Access and Mobility management Function

AN Access Network

API Application Programming Interface

ARP Allocation and Retention Priority

ARQ Automatic Repeat Request

AT3S Access Traffic Steering, Switching and Splitting

AUSF Authentication Server Function

BT3S Backhaul Traffic Steering, Switching, and Splitting

CN Core Network

CP Control Plane

CoS Classes of Service

DDCP Datagram Congestion Control Protocol

CRA Capacity Removal Algorithm

DN Data Network

DRM Decentralized Resource Manager

DS Differentiated Services

DSCP Differentiated Services Code Point

ECF Earliest Completion First

ECMP Equal-Cost Multi-Path

EIGP Extreme Interior Gateway Protocol

eMBB Enhanced Mobile Broadband

eMBMS evolved Multimedia Broadcast Multicast Service

eNB Evolved NodeB

EPC Evolved Packet Core

ETSI European Telecommunications Standards Institute

FEC Forward Error Correction

GBR Guaranteed Bit Rate

GEO Geostationary Earth Orbit

GFBR Guaranteed Flow Bit Rate

GMA Generic Multi-Access

gNB Next generation NodeB

GRE Generic Routing Encapsulation

HGF Hybrid Gateway Function

HoL Head of Line blocking

HTF Hybrid Terminal Function

HTTP HyperText Transfer Protocol

IETF Internet Engineering Task Force

IP Internet Protocol

IPSec IP Secure

ISP Internet Service Provider

LAN Local Area Network

LEO Low Earth Orbit

LOOPS Localized Optimizations on Path Segments

LTE Long Term Evolution

MA PDU Multi-Access PDU

MFBR Maximum Flow Bit Rate

M(V)NO Mobile (Virtual) Network Operator

MPQUIC Multi Path QUIC

MPTCP Multi Path TCP

SaT5G (761413) D4.3 May 2020

Acronym Description

MPA Multiple Path Algorithm

NAS Non-access Stratum

NAT Network Address Translation

NFV Network Function Virtualization

NFVO NFV Orchestrator

NSSF Network Slice Selection Function

N3IWF Non-3GPP InterWorking Function

OMSR On-demand Multipath Source Routing

OS Operating System

OSPF Open Shortest Path First

PCF Policy Control Function

PDU Packet Data Unit

PEP Performance Enhancement Proxy

PNF Physical Network Function

POI Point Οf Interest

POP Point Of Presence

PPP-MP Point-to-Point Protocol Multilink Protocol

PSA PDU Session Anchor

PSBOL Path Selection based on Object Lengths

QFI 5G QoS Flow Identifier

QUIC Quick UDP Internet Connection

QoE Quality of Experience

QoS Quality of Service

RAN Radio Access Network

RIP Routing Information Protocol

RM Resource Manager

RLC Radio Link Control

RQA Reflective QoS Attribute

RRC Radio Resource Control

RTT Round Trip Time

SaT5G Satellite and Terrestrial Network for 5G

Satcom Satellite communications

SDF Service Data Flow

SDN Software-Defined Networking

SMF Session Management Function

SNO Satellite Network Operator

SSH Secure Shell

STMS Slide Together Multipath Scheduler

TCP Transmission Control Protocol

TFCP Traffic Flow Control Protocol

TLS Transport Layer Security

TN Transport Network

UDM Unified Data Management

UDP User Datagram Protocol

UDR User Data Repository

UE User Equipment

UP User Plan

UPF User Plan Function

URSP UE Route Selection Policy

VIM Virtualized Infrastructure Manager

VNF Virtual Network Function

VNFM VNF Manager

VSAT Very Small Aperture Terminal

WLAN Wireless Local-Area Network

WP Work Package

WRR Weighted Round Robin

SaT5G (761413) D4.3 May 2020

Executive Summary

SaT5G aims at delivering the seamless integration of satellite into 5G networks and to ensure ubiquitous 5G access everywhere. This document corresponds to the output of SaT5G WP4.3 titled “Multi-link and Heterogeneous Transport - Analysis, Design and Proof of Concepts".

This document first analyses possible link aggregation protocols evolution and adaptability to satellite links at backhaul level. Satellite high latency is a strong challenge for 5G. Satellite Performance Enhancement Proxy technology is reviewed before recent multi-link protocols such as MPTCP and MPQUIC, including EC FP7 BATS’ multi-link (satellite / terrestrial) transport protocols. While multi-link use in backhaul is out of 3GPP’ scope, this document studies 3GPP multi-access AT3S standards and backhaul capacity to integrate a high latency satellite link to deliver a competitive user experience to 5G backhaul users compared to terrestrial only 5G.

This is mapped to SaT5G architectures as defined in D3.1 and D3.2, resulting in definitions of elements of the overall SaT5G architecture and the hybrid backhauling function BT3S, an extension of 3GPP AT3S for the backhaul supporting 5G AT3S UE as well as 5G non AT3S UE and non 5G hosts – leading to a contribution from SaT5G to 3GPP.

Study shows that using traffic splitting on multilink networks using satellite links along with low delay terrestrial links can solve the above issue provided that the appropriate path selection algorithms are used taking in account the size of the exchanged objects in order to steer the packets belonging to short objects on low delay/terrestrial links and to split the packets belonging to long objects on all the links.

Finally, two 5G multilink backhauling performance enhancement prototypes demonstrating MPTCP and MPQUIC sets of Transport Protocol & Link Aggregation Functions are detailed with the relevant results confirming satellite is eligible to complement a 5G backhaul.

SaT5G (761413) D4.3 May 2020

1 Introduction

5G is built on the assumption of unlimited bandwidth and sub millisecond latency. While this may be true for the urban population, the introduction and global roll out of eMBB service within 5G raises coverage and network dimensioning issues in underserved and unserved areas, especially in low ARPU regions of emerging markets, meeting/concert places where the number of users significantly increase only during few hours per month and on mobile platforms (e.g. vessels and aircraft).

Satcom systems are the only economic solution to address these scenarios by bringing extra bandwidth provided that they are seamlessly integrated into the future 5G architecture and optimum efficiency is achieved via technological synergies between 5G mobile and Satcom systems.

The SaT5G project aims to deliver the seamless integration of satellite into 5G networks and to ensure ubiquitous 5G access everywhere. Our aim through this project is to integrate satellite communications to the 5G infrastructure by investigating multi-links communication. SaT5G brings satellite communication (Satcom) into 5G by defining optimal satellite-based backhaul and traffic offloading solutions. It researches, develops and validates key 5G technologies in order to take the best value of Satcom capabilities (e.g. multicast for content and VNF delivery, ubiquity and resiliency) and mitigate its inherent constraints, e.g. latency.

SaT5G capitalizes on and drove the standardization effort initiated in 3GPP and ETSI since Q3 2015 by several consortium partners and Advisory Board members. A key feature of SaT5G is demonstrations of satellite integration in 5G network testbeds to validate the technology developed and scenarios. The project aims to be the main vector for defining the integration of satellite solutions for 5G in 3GPP.

SaT5G demonstrates satellite links integrated into 5G networks. These demonstrations includes radio resource management, satellite QoS integration and multi-link backhauling.

This document corresponds to the output of SaT5G WP4.3 titled “Multi-link and Heterogeneous Transport - Analysis, Design and Proof of Concepts", which defines how satellite specifics, such as radio and links can be seamlessly integrated into the Enhanced Mobile Broadband (eMBB).

As illustrated by Figure 1-1, the output of the Integrated network architecture design WP3 and especially the Reference Architecture D3.1 and Detailed Architecture D3.2 deliverables have described the design of an hybrid terrestrial and satellite system that is used as the context for this deliverable research. The output, especially the AT3S study and the multi-link prototypes built for WP4 are built, validated and demonstrated in WP5.3 on University of Surrey’ 5GIC network.

SaT5G (761413) D4.3 May 2020

WP6 Dissemination standards and exploitation

WP5 Validation and demonstrationWP4 Research to prototype developmentWP3 Integrated network architecture design

WP3.1 Reference architecture

WP3.2 Backhaul architectures

WP3.3 Cache & m/c architectures

WP3.5 Sat & 3GPP ref interfaces

WP3.4 End to end service delivery

Ref

ere

nce

arch

ite

ctur

e

D3.1 Reference architecture

D3.2 Detailed architecture

D3.3 E2E service delivery

D3.4 Sat & 3GPP reference interface

WP4.1 5G SDN & NFV in satcom

WP4.2 Integrated MANO

WP4.3 Multilink transport

WP4.4 Harmonising user & control

planes

WP4.5 Extending 5G security to satcom

WP4.6 Caching and multicast for

content and NFV

D4.1 Virtualisation of satcom

components

D4.2 Integrated MANO

D4.3 Multilink transport

D4.4 Control/user plane

harmonisation

D4.5 Extending 5G security to satcom

D4.6 Caching and multicast for

content and NFV

WP5.1 Demo planning and

logistics

WP5.3 5G plug n play demo

W5.2 Lab prototyping &

integration

WP5.4 Moving platform validation

WP5.5 User & control plane

validation

WP5.6 Results and recommendations

D5.1 Test bed plans

D5.2 Lab test integration &

validation report

D5.3 Fixed demo report

D5.4 Moving platform validation

report

D5.5 Plane harmonisations

validation report

D5.6 Results and recs from validation

and demos

WP6.1 Roadmap development

WP6.2 Standardisation

WP6.3 Dissemination

WP6.4 Exploitation plan

Findings and contributions to standards

Figure 1-1 Inter-relationships with other works in SaT5G

This document is organized as follows:

Section 2 analyses possible link aggregation protocols evolution and adaptability to satellite links at backhaul level. Satellite high latency being a strong challenge for 5G, satellite Performance Enhancement Proxy technology is reviewed before recent multi-link protocols such as MPTCP and MPQUIC, including EC FP7 BATS’ multi-link (satellite / terrestrial) transport protocols. Section 3 studies 3GPP multi-access AT3S standards and its backhaul capacities to integrate a high latency satellite link to deliver a competitive user experience to 5G backhaul users compared to terrestrial only 5G, it introduces BT3S. Section 4 investigates QoS in Satcom versus 3GPP. Section 5 presents 5G multilink backhauling performance enhancement prototypes demonstrating MPTCP and MPQUIC sets of Transport Protocol & Link Aggregation Functions. In section 6 we will list the standardization actions that occurred related on multi-link over an hybrid backhaul. Finally, we summarize our findings and give recommendations for future research.

SaT5G (761413) D4.3 May 2020

2 Multi-access State-of-the-Art

One of the defining 5G KPIs is to supply a higher throughput for subscribers to provide a better customer experience. This KPI is challenged when customers are provided with low bitrate 5G access. Bonding of satellite and 3GPP access networks can provide one such solution.

Performance Enhancing Proxies (PEP) are typical component found on satellite networks to address the system’s latency to be described.

The Broadband Forum (BBF) initiated architectural work on multi-link access that was finalized in 2016 [1] to define the business needs and use cases for hybrid access connectivity services. An architectural framework to realize them and the different deployment options were described. Ekinops had contributed to WT-438 to include a use case where the links would be different enough to include satellite. In the meantime the work on the protocols was progressing within [2] and [3].

The main objective of multi-access technology is to take advantage of the availability of multiple paths to improve the Quality of Experience (QoE) for end users. Improving the QoE means lowering the time spent in the network to make exchanges of information. When a satellite link is among the access links, its higher latency should be mitigated by introducing specific methods.

For network services where Packet Data Units (PDUs), i.e. objects are transmitted frequently in each direction, the latency and bandwidth are keys. Indeed, the transmission of short objects in a downlink direction for example is sensitive to the end-to-end latency on the path used and well suited on a low latency path. While the transmission of long objects is sensitive to the available bandwidth of the path, which is favourable on paths with high bandwidth. This performance can be improved by aggregating the bandwidth of several paths, and the latency should be the one from the lowest latency link. Path bandwidth and latency are meant as the ones of the worst link on the path.

To better understand how we can explore the multi-link network capabilities and protocols and how to apply that in a 5G context using an hybrid backhaul (including, satellite and terrestrial), we need to categorize the flows transiting over the network and review the mechanisms that are used for path selection. One of the objectives of the Sat5G system is to be able to use simultaneously several different networks (including satellite and terrestrial) in order to provide and consume network services. Such a multi-network infrastructure is heterogeneous. Therefore, considering the different types of flows is major.

2.1 PEP Performance Enhancing Proxies (PEPs) are network agents designed to improve the end-to-end performance of some communication protocols. PEP standards are defined in RFC 3135 [4] and RFC 3449 [5].

If one considers TCP the default maximum window size is 65,535B (64kB), this means that the server can send a maximum of 64kB before receiving an acknowledgment for the receipt of data. Given a typical Geostationary Earth Orbit (GEO) satellite link with a Round Trip Time (RTT) in the order of 650ms this means a maximum speed of 64k x 8 / 0.65 = 790kbps which is clearly too slow for modern broadband expectations. Consider a stream of 512B TCP datagrams this equates to ~200packets per second (pps); each data packet being acknowledged (ACKed) by 64B packets, this equates to 100kbps ACK traffic on the return link. This level of traffic is manageable in modern broadband satellite networks however may add significant cost overheads in some cases.

Modern TCP implementations allow window scaling which if both client and server support can increase the window exponentially to be 1GB. This does allow much higher transmission rates with no increase in ACK traffic. Window scaling can be disrupted by firewalls and similar processes however is enabled by default for most OSes, generally reasonably robust and well-documented online.

TCP PEP can implement the following tools to reduce traffic:

Terminate the TCP sessions at the satellite gateway and terminal. o This increases the initial handshakes and fast start-up processes,

Implements large window over satellite link to speed throughput,

Implements ACK aggregation to reduce ACK traffic.

This process is illustrated in [6] and below in Figure 2-1.

SaT5G (761413) D4.3 May 2020

Figure 2-1 TCP PEP in action

Obviously this can only work when the TCP headers are sent in the clear which is not the case when a L3 VPN such as IPSec is used, encrypted links using TLS that encrypt the data within the TCP datagram are generally OK.

This TCP layer PEP is often supplemented by one or more of the following features:

Lossless header compression (particularly useful for data sent in multiple small packets);

Lossless datagram compression (where the data is compressible and not encrypted);

Domain Name Server (DNS) caching (however many OSes and browsers implement their own DNS caching);

Pre-fetch web object caching (where the web page is parsed for objects and these are pre-cached by the system, this does not work for https web content however most modern browsers do their one pre-fetching).

This multi layering PEP is described in the relevant RFC 3135 [4] that differentiates these between transport layer and application layer. There is also a European Telecommunications Standards Institute (ETSI) technical report (ETSI TR 102 676) [7] that includes a discussion how distributing the PEP function can provide interworking with IPSec.

For PEP deployment, all modern consumer broadband VSAT modems and their hubs incorporate PEP either for the forward direction (download) only or both-way. The same technology albeit more powerful is included in the professional systems.

Many modern SCPC IP also include PEP albeit with a variety of marketing names (e.g. Comtech’s “TurboIP”).

There are acceleration appliances that can be deployed at each of a link to provide acceleration over links that include a Satcom element. This technology has been used for accelerating traffic within secure end-to-end VPN connections.

Software and VM implementations of TCP PEP are also available from different vendors.

The same kind of applications has been used for load balancing and similar applications; and for some terrestrial radio link uses.

SaT5G (761413) D4.3 May 2020

2.2 Type of flows A flow is classified according to the sequence of objects sent on a transport flow, e.g. a TCP/UDP connection or a Stream inside an HyperText Transfer Protocol (HTTP) 2/TCP or Quick UDP Internet Connection (QUIC), either as:

Interactive: short objects are sent in each direction. An example can be the Secure Shell (SSH) used to log into a remote machine and execute commands, another example could be the Terminal Network (Telnet) used in Local Area Network (LAN) to provide a bidirectional interactive text-oriented communication facility using a virtual terminal connection. User data is interspersed in-band with Telnet control information in an 8-bit byte oriented data connection over the TCP.

Download/Upload: short objects are sent in one direction, while the long objects in the other direction. It is typically the case of Cloud gaming, wherein gamers sends via their devices input commands (short objects) to the game engine hosted in the Cloud that compute these commands and then respond the gamer with a video stream (which are long objects). The TV and video dominant traffic falls in this category,

Mixed: the two above combined over the time: the flow is sometimes interactive sometimes in Download/Upload fashion. A prime example could be the Mobile Cloud Computing, where signalling messages (in general short objects) are exchanged between the User Equipment (UE) and the Control Plane (CP) hosted by a Mobile (Virtual) Network Operator (M(V)NO). This allows to authenticate the UE and enable the access to a remote service hosted in the Cloud. The service in general receives short object requests and responds with a long response. Another example of mixed traffic could be a Citrix remote user filling a spreadsheet who would generate small objects while another user printing a document would generate large objects.

2.3 Path Selection algorithms Now, we move our attention to the multi-link operation applicable on a path selection algorithm for outgoing packets. According to 3rd Generation Partnership Project (3GPP) [8], the standardisation of 5G network technology is underway to support Access Traffic Steering, Switching, and Splitting (AT3S) between 3GPP access network and non-3GPP access network.

Steering: traffic steering is the procedure that selects an access network for a new data flow and transfers the traffic of this data flow over the selected access network, This means all packets of the flow are sent over the same Link/Path as the first one (e.g. the least loaded or best performance link),

Switching: traffic switching is the procedure that moves all traffic of an ongoing data flow from one access network to another access network in a way that maintains the continuity of the data flow. Therefore, during its life a flow can switch on different links,

Splitting: traffic splitting is the procedure that splits the traffic of a data flow across multiple access networks. That is to say, some traffic of the data flow is transferred via one access network and some other traffic of the same flow is transferred via another access network.

Several scheduling algorithms have been designed for traffic splitting across multiple links based on different parameters. We discuss in the following some of these algorithms.

Lowest-RTT-First: this is the scheduling algorithm implemented in the Linux kernel. When the traffic arrives to the send buffer of the kernel, the first packet is always sent through the lowest RTT available path. This is a simple splitting policy, however it is not always efficient. Indeed, authors in [9] have shown that waiting for the fastest path to become available could be more efficient than sending immediately a packet on the slowest path if the difference between the RTT of these two paths is large enough,

Weighted round robin (WRR): the packets are cyclically balanced across the available links according to the links assigned weights in terms of bandwidth percentage, cost or else. Kae Won Choi et al. [10] have designed a weighted round-robin scheduler in a combination with load balancing for MPTCP-based bandwidth aggregation in heterogeneous wireless environments. WRR’s main advantage is the bandwidth consolidation, it is efficient as long as all the links have similar latency. And produces rather un predictable results when links are as

SaT5G (761413) D4.3 May 2020

different in latency than terrestrial and satellite links since packets are received in the order sent,

Offload: the arriving packets are sent on the same link as long as there is available bandwidth on this link. When there is not enough available bandwidth on this link, the packet and the following ones will be sent on another link (as long as bandwidth is available) [11]. This requires to measure the bandwidth and/or latency of all links,

Path Selection based on Object Lengths (PSBOL): an output of FP7 BATS [12] project, the PSBOL scheduler sends short objects or PDUs on the link with the lowest latency, while long objects are sent on the link with the highest available bandwidth. The classification of the objects is based on their size for which we define a predicate SIZE. We define a threshold for this predicate, then for all the objects exceeding this threshold, they are considered as long objects, otherwise, they are classified as short ones,

PSBOL combined with Offload: the PSBOL policy is applied first to distinguish the traffic. The short objects will therefore be sent on the lowest latency link/path while the long objects will be scheduled in offload fashion considering all the links/paths. This translates by filling the lowest latency link with short objects as well as long objects, and offloading the long objects traffic to the link with the highest bandwidth. This has been investigated within the H2020 VITAL project [13] for multi access with links which are very different, i.e. including satellite,

Earliest Completion First: several parameters are taken into consideration for this policy such as the RTT estimation, sub flow bandwidths (i.e., congestion windows) and the amount of data available to send (i.e., the send buffer). This policy considers the completion time as its main objective; the policy checks whether using a slow path for the injected traffic will cause fastest paths to become idle, so that the scheduler more efficiently utilizes the faster paths, maximizes throughput, minimizes download time, and reduces out-of-order packet delivery [14].

2.4 Multipath Methods and protocols Through this section, our focus will be on the protocols and methods that are unveiled in the literature for multipath communication. We will focus more on the technologies and proposals for hybrid satellite and terrestrial systems. ETSI in the document TR 103 124 [15] has given some reasons why we need to combine satellite and terrestrial systems. In the same document, ETSI had shared some associated scenarios. Including the satellite communications in some specific areas such as rural and hostile environments will allow service providers to rapidly deploy their services, increase service resilience through having a back-up path, as well as the total used bandwidth. The interactions between the satellite and terrestrial networks may take place at different levels, including:

Terminal: access to one component (e.g. backhaul scenarios) or to both types of components (e.g., integrated, hybrid or dual mode scenarios),

Spectrum: possible sharing of the same spectrum chunk by both satellite and terrestrial components (same or distinct frequency bands),

Network: pooling of satellite and terrestrial components resources or cascading the components (e.g. backhaul),

Service level: unifying service delivery via both satellite and terrestrial components,

Application level: defining new applications that take advantage of the specifics of satellite and terrestrial networks.

Multipath can be done at Layer 3 (IP), with or without tunnelling, or at layer 4. In the following, we will discuss the multipath splitting in these layers.

2.4.1 Layer 3 Employing several paths simultaneously has emerged as an efficient way to: (i) improve the throughput by aggregating bandwidths of multiple paths, (ii) increase the fault tolerance by duplicating the traffic over multiple paths, (iii) prevent congestions as the traffic is balanced across the chosen paths, and finally (iv) reduce the delay. This is especially true for technologies that are not subject to interference such as Ethernet, WLAN, and Long Term Evolution (LTE). It has been proved that the number of paths that can be selected as multipath routing is tightly correlated with the number of network technologies [16]. The multipath routing topic has been widely studied in multiple contexts including the Mobile Ad-hoc Networks (MANETs) [17],Wireless Sensor Networks (WSNs) [18], mesh networks [19] and traffic

SaT5G (761413) D4.3 May 2020

engineering [19]. Through this section we are more interested into mesh networks and traffic engineering based multipath routing.

In multipath routing, two topics should be considered, namely: the computation of multiple free paths and the split of traffic over these paths. Path quantity and path independence are two central characteristics used to determine a path set. The path quantity represents the number of free and available paths between nodes. The higher the number of paths, the better will be the load balancing. The independence between paths corresponds to the existing of disjoint links in these paths.

Currently, the most used routing protocols such as Routing Information Protocol (RIP), Open Shortest Path First (OSPF) or Extreme Interior Gateway Protocol (EIGP) are mono-path, hence the use of the bandwidth is not optimal. In the following we present several routing protocols discussed in the state-of-the-art multipath routing.

Equal-Cost Multi-Path: the Equal-Cost Multi-Path (ECMP) [20] is a routing mechanism of IP packets through multiple paths of equal cost simultaneously. The traffic is equally distributed over the equal-cost paths using Round-Robin scheduling algorithm. The ECMP feature was introduced to exploit shortest-path diversity by enabling the “split” of traffic between multiple shortest-paths via per-flow static hashing. This protocol provides several advantages: stable and predictable paths, relatively low protocol overhead, implementation in existing hardware, simple configuration language, scalability, and a built-in failure recovery mechanism to name few,

Multiple Path Algorithm: the Multiple Path Algorithm (MPA) was presented in [21] as an extension of the OSPF. The MPA algorithms search for subset of paths that satisfy the condition of loop-freeness. Each router considers only the paths with the next hop that satisfy the weight condition: the weight of the shortest path from the next hop to the destination should be smaller than the weight of the shortest path from the router to the destination,

Capacity Removal Algorithm: the Capacity Removal Algorithm (CRA) [22] is used to maximize the throughput. The algorithm considers real traffic varying conditions. The framework search for paths that increase the flow between nodes. It calculates the successive shortest paths, then decreases the path capacity (the minimal capacity of all links) from every link. A threshold link capacity is used to discard from the list of links those with the capacity is smaller than this threshold.

Several other algorithms have been discussed in the literature, we may cite the Multipath Distance Vector Algorithm [23], Multipath Partial Dissemination Algorithm [24], and Quality Multiple Partial dissemination Algorithm [25].

How packets are split among multiple paths is a major topic in multipath routing. Three main approaches have been discussed in the literature [26]: Modulo-N Hash, Hash-Threshold, and Highest Random Weight. In Modulo-N Hash, the next hop is selected from the list of N next hops. Indeed, the router performs a modulo-N hash over the packet header fields that identify a flow. The Hash-Threshold is performed in two steps: in the first step, the router performs a hash over the packet header to obtain a key which is assigned to a region. The N next hops have been classified into regions, so in the second step, the router compares the obtained key to the regions number in order to find the next hop. While in the Highest Random Weight, the routers computes a key for each next hop performs a hash over both the packet header fields that identify the flow, and the address of the next hop. Then, the router chooses the next hop with the highest resulting key value.

Hash-Threshold algorithm is commonly used. This algorithm has been studied in [20]. It is stated that if the hash is done on the IP addresses of the source and destination using the CRC with 16 bits, so the traffic splitting is done in coarse granularity. The same operation leads to a finer granularity when using 32 bits CRC to hash the IP addresses and port numbers. However, performing a hash key at each hop using 32 bit CRC and TCP port numbers is expensive. In [27], authors proceed differently with the TCP/UDP packets. They propose a modified version of the hash-threshold scheme. A random key is generated and inserted in each packet using the source and destination IP addresses and port numbers. This process in done at the ingress router. Within the Core Network (CN), each router uses this key to determine the next-hop. The authors use the 13-bit fragment packet offset field to store the hash key (if fragment packet offset field is zero, i.e. the packet is un-fragmented). They also propose to use the TOS or DSFIELD bit to indicate whether the packet is a UDP or TCP packet. The advantage of this approach is that there is no need to read packet header at each hop and hence, facilitates faster forwarding, but fragmented packets arrive out-of-order.

SaT5G (761413) D4.3 May 2020

There have been recently, several studies on how to apply multipath routing to MPLS networks. These works are related to load balancing, traffic splitting, and fault tolerant. For traffic split ratio, several mechanisms and frameworks have been proposed. In [28], the authors have presented an analytical framework for dynamic traffic partitioning in MPLS networks. The aim was to minimize the end-to-end delay and the packet loss rate. The paper was missing to discuss how to find the appropriate number of multiple paths that should be used. The authors used a dynamic stochastic partitioning algorithm and assignment methodology to adaptively map ingress traffic into several parallel paths. The disjoint paths were modelled as parallel queues and the partitioning algorithm was devised for different class of services that is adapting to the prevailing state of the network. Yongho Seok, et al [29] have presented a dynamic constrained multipath routing framework for MPLS Networks. The paper presented an heuristic algorithm for hop-count and path-count constrained dynamic multipath routing. The aim was to minimize the maximum of link utilization and obtain a ratio for traffic split among the different paths by using a hash method at flow level. Another framework was presented in [30] for constrained multipath traffic engineering in MPLS networks. The paper proposes a mixed integer linear programming formulation to calculate a pseudo-optimal paths satisfying the traffic demands under some constraints such as respecting a maximum number of hops, and preferred node (or links) list. It also calculates the traffic split ratio for multiple paths.

2.4.2 MPTCP MPTCP is a fully compatible extension of the TCP protocol, published as an experimental standard in 2011 [31], [32], [3] and currently widely deployed [33]. MPTCP is devised to allow customers consuming services on their device, which are embedded with multiple communication technologies, such as WLAN and 4th Generation mobile network (4G), to use multiple connections at the same time without changing the socket API. Since the first implementation [34], MPTCP to deliver fast Internet services on different but similar in latency links continues to be studied [35]. The aim is to increase both throughput and reliability, and reduce the network latency with the load on congested paths. In order to ensure compatibility and transparency to the services, MPTCP mechanism needs to be implemented at the operating system kernel, within the IP/TCP stack.

MPTCP operation is negotiated at the TCP connection establishment time (handshake connection) by setting the MP_CAPABLE option of the first TCP packet (0x30). The MPTCP session starts with one sub-flow. If the MPTCP operation is accepted by the remote host then, the two hosts exchange cryptographic keys in order to add new sub-flows securely, on each available path. These sub-flows are established by setting the MP_JOIN option and using the hash of the connection’s keys during the standard TCP handshake. Each MPTCP sub-flow is a TCP connection having its own sequence numbers and congestion management algorithm. The Payload of a sub-flow segment carries the payload of a segment of the original TCP connection depending on the Path Selection Algorithm used and is reliably transmitted thanks to the TCP nature of the sub-flow. Each sub-flow segment has its own sequence number and carries in the TCP Option (0x30) the sequence number of the corresponding segment in the original connection. This may allow reordering the packets when received even if re-transmission can be on another sub flow. Since the congestion control and retransmission is happening at the sub-flow level (because each sub-flow is a full TCP connection), the flow-level sequence numbers should be consecutive and the data on each sub-flow is delivered in order [xxx].

Retransmission of segments on a sub-flow can be done on another sub-flow. Similar to the sequence numbers, there are two types of ACKs: a regular acknowledgment for each TCP sub-flow, and a connection-level ACKs as a cumulative acknowledgment for the entire data flow.

Scheduling in MPTCP is central. Indeed, sending data on the wrong path can exacerbate the Head of Line blocking (HoL) problem, increase the network delay or even lower the throughput. Currently, the scheduler in the Linux kernel follows the Lowest-RTT-First (LowRTT) policy [36], wherein each packet is sent on the fastest available path. Other scheduling policies have been developed, we may cite the Weighted Round Robin scheduler [37], [10], the Earliest Completion First (ECF) algorithm [38], Slide Together Multipath Scheduler (STMS) [39].

The FP7 BATS project added Packet Steering based on Object Length (PSBOL) scheduling where objects below a given size, considered as short, as sent over the lowest latency link while objects longer are sent over the highest bandwidth link. PSBOL and Offload can be used as Path Selection algorithms as demonstrated in H2020 VITAL. MPTCP can be also considered in a tunnel mode. In this case the outgoing IP (TCP UDP) packets are sent in the payload of the MPTCP subflows.

SaT5G (761413) D4.3 May 2020

In the technical report 103 272 [40] published by ETSI in 2015, the focus was on proposal of an hybrid access network combining one or several terrestrial access technologies (Fixed or Mobile Service) together with a satellite broadband access network (Fixed Satellite Service) and discussing the architectural issues emerging from this proposal. In ETSI TR 103 351 [41], a deeper discussion on how to use the MPTCP for hybrid access network architecture.

Figure 2-2 Hybrid access using multiple paths between gateways [40]

ETSI have unveiled the hybrid architecture shown in Figure 2-2. It involves two gateways, one in the customer’s premises at the home gateway and one in the network. Between the two gateways multiple networks are used; the gateways hide the multiple networks from the end hosts, so it looks like a normal network but with better performance.

The gateways have several jobs such as

Signalling between gateways in order to set up the multiple connections - for instance with MPTCP, to add sub-flows to a connection - and remove them,

Deciding on how to spread traffic over the multiple sub-flows. For instance, many deployments of MPTCP send traffic over one sub-flow and keep the other(s) on ‘hot standby’,

Reacting to the links performance and traffic type accordingly (e.g., the unavailability of one path or due to its performance degradation, and putting traffic that needs low latency over one path),

Monitoring the performance of each access link in terms of throughput, latency and packer loss. Alternatively, this may be achieved by the packet forwarding process,

Discovering the network operator policies, and considering them into the scheduling (such as, link prices and customers preferences, and operator trust),

Splitting the traffic from a single connection over multiple sub-flows, and merging traffic from multiple sub-flows into a single one (i.e., converting between TCP and MPTCP),

When it comes to decide how to spread traffic there is a trade-off between service efficiency and other factors such as technical difficulty. The transmission resources are most efficiently used by deciding packet-by-packet which access to use, essentially by using TCP’s feedback mechanism to choose the path with the lowest congestion. However, this requires buffering packets in order to even out differences in RTT and to handle re-transmissions of lost packets.

SaT5G (761413) D4.3 May 2020

Hence a common compromise is for each traffic flow to use a single path, i.e. to spread traffic flow-by-flow rather than packet-by-packet. Further issues should be considered including the access networks with different RTTs and traffic classification. Regarding the traffic classification, ETSI in [41] distinguished two types of traffic; the critical and non-critical traffic and suggested to treat these traffic differently based on the link reliability.

Whilst the above aspects concern the data and control planes, the management plane needs also to be adjusted. Indeed, we may have several options for the commercial relationships between the involved actors. In ETSI TR 103 272 [40] we have identified three options namely: (i) The Internet Service Provider (ISP) requests for satellite service from a satellite provider which also runs the network-side gateway; (ii) The ISP operates the satellite service and requests for a terrestrial service from a network operator; (iii) The ISP requests for services from a wholesale provider, which in turn requests these services from terrestrial and satellite providers). Other options may emerge from this ecosystem such as, an ISP requesting for the satellite service from a satellite service provider while using his own gateway. Another point is about coordinating the orchestrators. Indeed, the ‘softening’ of networks (Software Defined Networking (SDN), Network Function Virtualization (NFV) and future similar developments) means that network capabilities and services can be much more flexible and dynamic. Yet different aspects are run by different parties, who may change their vendor, upgrade their capability, offer new services or develop new commercial relationships with each other. Each party uses its own orchestrator layer that may include the NFV orchestrator (NFVO), the Virtual Network Function Manager (VNFM), slice manager, and SDN controller. For example, OSM defines an NFVO including the VNFM (Juju) and potentially the slice manager. Those multiple orchestrators need to be involved and be coordinated together to deliver the service.

Although not mentioned in the ETSI document, it is also possible for the multiple connections to run to one or both end points. Multi-connected mobile devices are now very widespread, typically with 3rd Generation Mobile Network (3G), 4G, and WLAN, and there are many deployments of MPTCP on such smartphones. For instance, all Apple iOS devices use MPTCP for Siri [42] [43] and there is an open API so that other application developers can use the MPTCP.

2.4.2.1 MPTCP Hybrid solution & proxy At the time of standardizing the MPTCP protocol, the IETF has considered some aspects of how to implement a hybrid solution. In scenarios where either one or both end points are not MPTCP enabled, then a gateway can convert between TCP and MPTCP. The IETF has considered two options whether or not the proxy is implicit or explicit: When the gateway acts as an implicit proxy, it converts in transparent manner the TCP connections established by the attached clients into MPTCP connections. In its role as an explicit proxy, the gateway is a TCP end point. The end-user explicitly asks the proxy to terminate the TCP connection and create a MPTCP connection to the other end point or gateway.

Other approaches for instance where there are tunnels between the gateways have the problem that there are two independent congestion control loops, one operating end-to-end and one between the gateways. Experience has shown this can lead to problems with the control loops being in competition, leading to system instability,

The IETF SDO is in a process standardization of the explicit proxy approach as “0-RTT converter”. The main reason for choosing this approach were worries about breaking the end-to-end principle. Most deployments so far have used an implicit proxy.

2.4.2.2 Non-TCP traffic While most traffic is TCP-carried, there are however, approaches to transport non-TCP traffic including the following:

Forward the traffic over a single path. because conventionally UDP traffic does not benefit from multiple paths since it tends to be short messages and/or highly latency sensitive, so it is negatively impacted by the multi-path overhead, and/or the added latency occurring when waiting for packets over a longer latency path. Also, some applications may have application-level multipath, and people have proposed multipath RTP,

Encapsulate the traffic in TCP at the gateway: encapsulating the traffic in a TCP at the gateway provides the way to handle this traffic as MPTCP. This re-uses the MPTCP solution but adds multipath to applications that may not need it. Considering the UDP as datagram-oriented and

SaT5G (761413) D4.3 May 2020

TCP as stream-oriented, so packet boundaries need to be maintained when tunnelling over TCP. TCP also adds reliability, which may not be a useful when UDP is used for timeliness (e.g. by RTP). Other IETF transport protocols that are capable of using multiple paths include SCTP [44], MPTCP [31] and SHIM6 [44]https://tools.ietf.org/html/rfc5533. However, these protocols are not suitable for real-time communications,

Use MPQUIC: QUIC is a UDP-based protocol proposed by Google as an alternative to improve TCP to increase traffic throughput by shortening connection setup and leverage UDP’s connectionless nature. Since its inception in 2012, QUIC has been growing its market share to be supported by more than 11% of servers traffic [45], which its proponents hope will take over from TCP’ domination. The IETF’s QUIC Working Group plans to standardize MPQUIC, starting late 2020. The process can learn from MPTCP, so could take 2 years. A key point about QUIC is that the transport layer is encrypted by the end hosts, including metadata. This seems to make any implicit approach harder (as no information about the traffic is visible, except the destination and source addresses),

Create a multipath UDP: this approach could be instead of the previous one, i.e. handle QUIC traffic (which looks like UDP on the wire) as well as ‘ordinary’ UDP traffic. One interesting current proposal uses Datagram Congestion Control Protocol (DCCP). DCCP was standardized as a kind of UDP replacement, but has seen limited deployment.

2.4.2.3 Practical work on MPTCP and satellite systems There are some limited public discussions on how to practically use the MPTCP within satellite systems. The NASA had presented some work at IETF-93-interim & IETF-95 on using MPTCP to bond multiple channels for airborne science missions. The system essentially has four Iridium Satcom channels at 2.4 kbps (9600 bps total) above 65 degrees latitude (NASDAT, NASA Airborne Science Data Acquisition and Transmission, Global Hawk Network). The default approach is to use Point-to-Point Protocol Multilink Protocol (PPP-MP), which fragments TCP (and UDP), which causes problems if any of the modems goes down, as the communications tends to block and take a long time to detect the problem. MPTCP gave better performance. They experimented with different MTUs, the small one have provided better performance.

CNES, the French National Centre for Space Studies, had presented a work at IETF about the performance of MPTCP over satellite for broadband access [46]. A satellite system includes a Performance Enhancing Proxy (PEP). The PEP splits the TCP connection into several segments each with a short RTT, which improves the overall TCP throughput as this is proportional to 1/RTT. The PEP is responsible for (i) adjusting the way acknowledgments are done (that includes sending acks optimistically, i.e., the PEP sends an ack to the source as soon as it gets the TCP packet, instead of the ack coming from the receiving end host); (ii) optimizing the window size for each, so that interim TCP senders do not need to wait for data, and so that transmission errors over the satellite can be fixed with a local re-transmission (by default TCP assumes that errors are caused by congestion and decreases its rate accordingly); (iii) and potentially other similar techniques to deal with variable RTT and asymmetric bandwidth. The CNES experiments have shown that an MPTCP-enabled PEP successfully exploit both the satellite and 4G paths (Evolved Packet Core (EPC) with the Radio Access Network (RAN)), so that it delivered improved performance, except for small files. Results were also better where MPTCP ran end-to-end and without PEPs in the satellite link. Unfortunately, the experiments did not directly compare MPTCP in the PEPs vs end-to-end.

Authors have simulated a Low Earth Orbit (LEO) satellite network carried on MPTCP. The authors were motivated by the use of MPTCP for LEO satellite network because of the improvements that MPTCP may provide in terms of increasing the bandwidth and compensating the impacts of the longer RTT during handover. The authors have classified the transmission state into stable and handover states. The stable state corresponds to the period of time during which the gateway satellite is constantly visible to the ground terminal, while the handover state refers to the satellite handover. As LEO satellite constellation is a mesh network, thus during the stable state, the traffic can be split on different MPTCP subflows into satellite-to-satellite paths. Whilst, during the handover state, the ground terminal can set up connections with different gateway satellites, as it is under the coverage of multiple satellites. The authors have proposed an On-demand Multipath Source Routing (OMSR) algorithm that builds multiple disjoint paths between satellite pairs on three steps. First, the algorithm detects as much as possible of disjoint LEO satellite paths. Second, checks each of the detected paths, weather it is usable or not based on defined path selection rules. Finally, forwards the packets along the usable paths. For their

SaT5G (761413) D4.3 May 2020

tests, the authors have simulated the Iridium constellation to represent the LEO satellite networks and used the NS2. A comparison between the MPTCP-OMSR and two variant of TCP has been performed. The MPTCP-OMSR provides one order of magnitude better performances in term of throughput and delay than either TCP flavour.

Authors have considered the MPTCP in LEO satellite networks which are controlled by a SDN controller. The authors have designed an SDN controller that identifies MPTCP subflows attached to the same MPTCP session and split them to disjoint paths. The authors have tried to tackle with the drawbacks of their previous solution (the MPTCP-OMSR). Indeed, as the OMSR is a distributed routing protocol, it introduces overhead in the path discovery operation. In addition, because the OMSR runs on several satellite gateways independently, so when a ground terminal connects to two different satellite gateways during a handover, it may fail to split the MPTCP subflows to disjoint paths due to the lack of inter-satellite communications. To improve the solution, the authors have centralized the routing intelligence of the LEO satellite network using an SDN controller. The assumption regarding satellites turned into SDN switches and executing simple forwarding rules had been considered here. The authors have used the Mininet emulator to simulate the network. The results improve the throughput significantly and prevent from transmission interruption during handovers.

2.4.3 MPQUIC

2.4.3.1 QUIC The recent trends on how services are designed have pushed for the need to revise the network communications especially, with the imminent arrival of the 5G. Indeed, this will drive changes in communication services, which require extra low latency, security and privacy of the user data, which even adds delay. Efforts to reduce such latency have been made on the transport mechanisms as one solution. While new protocols have been designed to meet the application demands in term of latency such as DCCP [47] and SCTP [48], they have not seen wide deployment [49] [50]. Indeed, as the middle boxes have become the control points in the network such as firewalls tend to block unfamiliar traffic for security raisons. Focusing on the well-known transport protocol TCP to deploy changes is not either the best option, as it is expected to take more than a decade to be significantly deployed. In addition, TCP is deeply embedded in the Operating System (OS) kernel, hence pushing changes to the TCP stack will require OS upgrade. This limits the deployment velocity of the TCP changes as the OS upgrade has a wide impact on the network. The other two limitations of the TCP are the handshake delay and the Head-of-line blocking. The handshake takes one RTT to setup a connection before an application can send data; when it is coupled with Transport Layer Security (TLS), it adds two RTT to this delay. Head-of-Line blocking occurs in case of packet loss and when packets are transmitted with long delay, which increases the overall delay.

To handle such issues, a new protocol called QUIC (Quick UDP Internet Connections) has been implemented. The QUIC protocol was introduced by Google in 2013 and currently considered for standardization by the IETF [51] [52]. It is an application layer transport mechanism, implementing TCP-like properties over UDP transport layer such as the congestion control and loss recovery as well as the acknowledgment mechanism. Besides, QUIC incorporates some other features including the TLS 1.3 key negotiation requiring the entire connection to be encrypted, and the HTTP/2 features like multi-streaming [53]. QUIC is not implemented in the kernel, but distributed either as a user-space library or in the application itself. It is carried over UDP transport protocol, mainly because working over UDP offers an easier migration route than trying to do major surgery on TCP. Besides that, the use of UDP allows QUIC packets to transit through middle-boxes without dropping. QUIC aims to be equivalent to TCP with better performance. It provides three major advantages over TCP [54]:

Reduce the handshake delay: most of the services require secure and reliable network

connectivity. It is the role of TCP and TLS to provide such requirements. QUIC is two

(respectively, one) RTT less than TCP coupled with TLS 1.2 (respectively, 1.3). With successful

negotiation, TCP coupled with TLS 1.2 (respectively, 1.3) requires three (respectively, two)

RTTs (i.e., one RTT for TCP handshake and two RTTs for TLS 1.2 handshake, or one RTT for

TCP handshake and one RTT for TLS 1.3 handshake). QUIC combines the transport and crypto

to minimize the connection latency, carrying both TLS handshake and the relevant QUIC

transport set up parameters in the first packet of the connection. The known server credentials

SaT5G (761413) D4.3 May 2020

negotiated during first connection will be reused in future connections, which reduces the

latency to zero-RTT connection establishment,

Eliminate the Head-of-line blocking: within TCP, a single connection carries all the

communications between a client and server, which can lead to head-of-line blocking (where

one lost packet blocks delivery of all data queued behind it until the packet is re-transmitted, as

the data must be delivered in order). QUIC eliminates this delay by using a lightweight data-

structuring abstraction, streams, which are multiplexed in a single connection; each stream

having independent data transfer and flow control, allowing different transactions to proceed in

parallel, so that loss of a single packet blocks only streams with data in that packet,

Enhance security and privacy: data and signalling packets in TCP are sent in the clear. By

contrast QUIC authenticates all of its headers and encrypts the data it exchanges, including its

signalling. It also makes it harder for middle-boxes to “bake in” assumptions about a protocol,

which has caused the ossification of several protocols (for instance, one reason why it is very

hard indeed to deploy widely any changes to TCP).

Note that modern, emerging TCP variants use the same techniques to get the same benefits, which reduces the incentive to switch to QUIC. TCP coupled with TLS 1.3 have the same mechanisms to reduce connection handshake. Techniques such as HTTP pipelining and Structured stream transport use the same multiplexing approach. TLS 1.3 uses enhanced encryption hence, QUIC essentially incorporates TLS 1.3, for key negotiation, confidentiality and integrity protection of both application data and QUIC headers.

On the other hand, there are potential issues with QUIC, which may slow its progress – although we can expect their impact to diminish if/when QUIC gets used more extensively. Indeed, it takes a network device more processing to compute a QUIC packet than a TCP one. Middle-boxes raise several questions: UDP is blocked by some perimeter protection firewalls; Network Address Translation (NAT) state is usually timed out quickly (meaning that the QUIC connection cannot be long); and they will not work with end-to-end encryption. Finally, QUIC’s encryption makes it harder for operators that rely on passive monitoring to make the necessary measurements and fault analysis.

The current suggestion is that a QUIC-enabled browser should initially use TCP to contact a webserver. The webserver can reply with “HTTP Alternative Services (Alt-Svc)” to indicate that the webserver is QUIC capable. The browser would then use QUIC for future requests. Another possibility may be to use a ‘happy eyeballs’ approach: the browser tries to contact the web server with both TCP and QUIC and sees which responds first (or gives QUIC a small head start).

Figure 2-3 depicts a comparison between the TCP and QUIC protocol stacks combined with the HTTP/2. QUIC incorporates TLS 1.3, which reduces the duration of the connection establishment and allows QUIC to traverse middle-boxes without having their inside information tampered with.

Figure 2-3 Comparison of TCP and QUIC protocol stacks

In fact, only the UDP header and selected fields of the QUIC header are not encrypted as shown in Figure 2-4: the red fields are encrypted and the blue ones (i.e., the packet number field in the header) are protected against casual observation. It also uses a multi streaming function to improve the TCP’s

SaT5G (761413) D4.3 May 2020

loss recovery by explicitly distinguishing in the ACKs between lost and out-of-order packets. A single connection is composed of multiple independent streams, so that the loss of a packet affects only one stream.

Figure 2-4 Structure of QUIC packet with long-header for IETF QUIC (version 14)

2.4.3.2 QUIC over satellite links There have been studies and experiments about how QUIC operates over satellite links [55]. These are still at an early stage, so there is not yet complete consensus on how it performs. An overall summary of the experiments is that the performance with (end-to-end) QUIC is better than the performance obtained with end-to-end TCP and somewhat worse than for TCP with a split-TCP-PEP, especially with larger files. We can expect that the gap will shrink as QUIC implementations mature and with new mechanisms.

The main challenges for a satellite-based system arise from the large propagation time, which gives a round trip time (RTT) for GEO systems of more than 600 milliseconds. hence:

Start-up is slow, due to the large RTT, both in terms of the initial handshaking and the congestion control’s ‘slow-start’ mechanism (exponential increase to probe the channel’s capacity),

A large window is required to exploit the capacity (due to the high bandwidth delay product). TCP’s current recommendation of 10 segments (with 14.6kB) is not enough with a satellite link,

Re-transmission of any lost packets is slow. This is the major motivation for Performance Enhancing Proxies in satellite systems. PEPs halve the time to load a webpage. PEPs enable “split TCP” meaning that rather than an end-to-end TCP connection, instead it is divided into three TCP connections (from the client to the first PEP; between the PEPs over the satellite link; and from the second PEP to the server). The satellite link itself has often very low packet loss ,due to its adaptive coding and modulation, which may increase on the end-to-end path where other access technologies are used such as WLAN,

Another important difference is that a satellite link is typically highly asymmetric, in terms of capacity. It is possible for the throughput to be limited because the rate of acknowledgments is constrained. PEPs can overcome this problem – cf. Figure 2-5,

TCP PEPs tune the TCP parameters to the specific latency and capacity of each leg,

TCP PEPs also hamper the usage of new TCP improvements, since they can only be used if they have been rolled out to the PEPs as well as the end hosts.

SaT5G (761413) D4.3 May 2020

Figure 2-5 Satellite system using split TCP with performance enhancing proxies

With QUIC there are two main differences:

QUIC’s reduced connection handshake is a significant advantage,

QUIC’s end-to-end encryption makes PEPs and so split-TCP impossible, which is a significant disadvantage.

Measurements are just starting looking at how QUIC operates over satellite links, and to try and assess the relative importance of these two factors. Summarizing the recent experiments:

Thomas et al. [56] have measured Google QUIC’s performance over a public GEO satellite communications network. For a large web page (over 5MB), the page load time is approximately twice as long with Google’s QUIC compared to split TCP (with TLS 1.2), mainly due to Google QUIC’s congestion controller spending most of the time in slow start; this degradation is not compensated by Google QUIC’s faster connection establishment. For a small webpage (11kB) the page download time was slightly faster with Google’s QUIC.

Mogildea et al. [57] compared the performance of three QUIC implementations (Google’s QUIC and two implementations of IETF QUIC) and TCP, both with and without a PEP, over three geostationary satellite networks offering from 16 to 30Mbps on the downlink and from 2 to 3Mbps on the uplink. The PEP-less experiments are created by using a VPN, whose end-to-end encryption prevents the satellite system’s PEP from working. With no packet losses, split TCP download of a 1 MB file takes less than 4s, whilst the median time with QUIC varies from 4 to 12s (depending on the QUIC variant and the satellite operator) – typical is about 8s- and worst case results are increased considerably more. The authors then simulated 1% packet loss rate (this might correspond to a Wi-Fi link on the end-to-end path). The results showed that none of the QUIC implementations could handle loss well, with two of the QUIC implementations typically taking more than 12s for the download, whilst the split TCP takes up to4s. The authors also speculate that the difference in QUIC performance between the satellite operators is mainly due to differences in UDP performance (differences in median one-way UDP delay and in range of delays).

Fairhurst et al. [58] have compared QUIC, TCP and Open-VPN over a high delay path. This was a satellite residential broadband connection. For a 1MB file, the median download times were about 4s for TCP (and TLS 1.3) with a PEP, 6.5s for QUIC and 15s for PEP-less TCP. For a 100kB file, the median download times were 3s for split TCP, less than 4s for QUIC and 5s for the VPN.

In [59], the authors have studied how a satellite link could be integrated to multi-link 5G standards. Two functions have been defined to allow multi-link operations on an hybrid backhaul network namely; Hybrid Terminal Function (HTF) and Hybrid Gateway Function (HGF). These two functions are in charge of managing each link of the hybrid backhaul and organize the multi-link splitting operations for AT3S (AT3S will be discussed in details latter on) enabled UEs, non-AT3S enabled UEs and regular non 5G hosts. The HTF is located at the edge, it can be a standalone equipment, or hosted in the gNodeB or the satellite terminal. While the HGF can be a standalone equipment or hosted in the UPuAT3S. For the traffic splitting, the authors have compared four algorithms: Weighted Round Robin (WRR), Path Selection Based On Object Length (PSBOL), Offload, and a mixture of PSBOL and Offload. The obtained results have shown that the combination of PSBOL and Offload outperforms the other algorithms.

Some ideas are being discussed about how to improve the performance of QUIC over satellite systems at the IETF and in the research community:

SaT5G (761413) D4.3 May 2020

Provide the QUIC end points with some awareness of the path’s properties, so it can optimize some parameters, such as increasing the initial window. In a more general context, these ideas are being discussed at the IETF in the Path Aware Networking research group.

On a related theme, improve QUIC so it can detect the unusual (end-to-end) path characteristics (arising from the satellite link) and adapt its parameters appropriately. For instance, to increase the maximum number of packets in flight (because of the larger bandwidth-delay-product) or to use compound acknowledgments (if throughput is limited by the rate of ACKs on the low bandwidth reverse link).

Optimize the network, whilst not involving the end point. For example, the packet loss rate could be lowered, since packet losses are particularly problematic. The two main techniques are to add to the problematic path segment either Forward Error Correction (FEC) or Automatic Repeat Request (ARQ). The IETF’s Localized Optimizations on Path Segments (LOOPS) working group is studying this in a slightly more general context. Other ideas could be to add FEC end-to-end or use network coding.

2.4.3.3 Multipath QUIC In summary, MPQUIC could potentially be deployed instead of MPTCP, although there are unresolved issues where the multiple paths do not run to the end hosts, which would be typical for a satellite system. We may mention two motivations for which QUIC is evolved to MPQUIC namely, to pool resources of different paths to transport the data for a single connection, and to make the communication more resilient to failures. Since QUIC is implemented in the user-space, its extension is much easier than the MPTCP which is implemented in the OS kernel. While the need to revise and improve how the transport protocol are used, the body of literature related to MPQUIC is surprisingly small with regards to the significant advantages that it may provide. In this regard, we may mention three position papers that propose a design and implementation of this protocol [60], [61], [62].

Work has begun to create a multipath version of QUIC running between end hosts that have been tested [63].

MPQUIC takes the same basic approach as MPTCP. The work is being done by the team at UCLouvain who were the main developers of MPTCP. Whilst MPQUIC is part of the IETF’s plans, it will not begin official standardization until the first version of QUIC is complete, roughly Spring 2020. At the moment the focus is on multiple paths that are locally visible to an endpoint.

An early comparison of MPTCP and MPQUIC, using measurements on an emulation platform, shows that MPQUIC performs somewhat better. Reasons include its better loss signalling, more precise latency estimation, and improved fairness in the evolution of the congestion window of paths – essentially, QUIC’s clean slate design allows us to avoid some of the sub-optimal aspects that were necessary to fit with the details of TCP.

However, QUIC’s end-to-end encryption prevents an implicit MPQUIC proxy, and it also seems to make an explicit MPQUIC proxy problematic. One proposal is to establish multiple Datagram Congestion Control Protocol (DCCP) tunnels to carry the QUIC traffic between the gateways. DCCP was standardized as a kind of UDP replacement, but has had little deployment.

SaT5G (761413) D4.3 May 2020

3 3GPP and Multi-Access

Multi-access is considered at 3GPP for the 5G architecture. The multi-access study at 3GPP has been started with the 4G, with focus on content distribution. There it was possible to have coordinated transmissions from two neighbouring evolved NodeBs (eNBs), and it was possible to have a UE listening to both of them at the same time. The idea was that at the edges of the cells, where the signal is weak, receiving the same signal from two different eNBs makes it possible to combine them on signal level, and therefore achieve gain, making successful reception possible. In that approach, UE maintains connection to the two eNBs, both with the same access technology, and the content was the same. One of the main additions in 5G, is the integration of different non-3GPP technologies (accesses) to its CN. While this was possible also in 3G and 4G, it is now possible in 5G to use mobility management across different access technologies, to use the same authentication mechanisms as in 5G.

One of the objectives of the 5G is to provide new radio, new 3GPP access technology. Therefore, it is possible to use existing access technologies, with the goal of providing increased throughput, robustness and mobility (across access technologies). In that light, the 3GPP consortium have described in the document TR 23.793 [8] how to enable multi-access to both 3GPP and non-3GPP technologies.

3.1 AT3S Function The SA2 3GPP working group is studying the multi-link aspects in 5G. The document [8] studies how the 5G System (5G UE and CN) can be extended to support Access Traffic Steering, Switching and Splitting (AT3S) between 3GPP and non-3GPP access networks.

Before going further, we remind here what means Access Traffic Steering, Switching, and Splitting as described in 3GPP:

Access Traffic Steering: The procedure that selects an access network for a new data flow and transfers the traffic of this data flow over the selected access network. Access traffic steering is applicable between 3GPP and non-3GPP accesses,

Access Traffic Switching: The procedure that moves all traffic of an ongoing data flow from one access network to another access network in a way that maintains the continuity of the data flow and service. Access traffic switching is applicable between 3GPP and non-3GPP accesses,

Access Traffic Splitting: The procedure that splits the traffic of a data flow across multiple access networks. When traffic splitting is applied to a data flow, some traffic of the data flow is transferred via one access and some other traffic of the same data flow is transferred via another access. Access traffic splitting is applicable between 3GPP and non-3GPP accesses.

The idea of multi-access is put on top of existing 3GPP CN architecture, with the goal of defining what elements should be adjusted and how. Note that 3GPP is focusing on access technologies, and not on the backhaul. The backhaul itself is another topic, as it is not in the scope for 3GPP however, with the introduction of satellites as a possible backhaul technology, the focus and interest of the 3GPP might change.

In the scope of SaT5G, where satellites are used for the backhaul, the main focus is on understanding the 3GPP mechanisms used for the AT3S. Then move towards mapping these mechanisms with the SaT5G architecture, including backhaul.

3.2 AT3S architecture framework Access architecture is an expansion of the existing 5G architecture. A number of existing network elements in the Core Network need to be adjusted, namely those handling the control and user planes. The AT3S architecture is focusing on the access part of the network. Proposed architecture can be found in the document TR 23.793 [8], and it is shown in Figure 3-1.

AT3S is initiated by the UE and terminated in the Core Network. applications in UE and the N6 host server are not aware of traffic split across multiple accesses.

The main assumption is that at least one access technology is 3GPP. It is the responsibility of the Non-3GPP InterWorking Function (N3IWF) to manage the connectivity between the 3GPP and non-3GPP technologies. The case of connectivity between two 3GPP technologies has been dealt with in Long Term Evolution (LTE) for evolved Multimedia Broadcast Multicast Service (eMBMS), where both

SaT5G (761413) D4.3 May 2020

accesses belong to the same operator. The case with multi-access via two (or more) different operators) has not been studied.

As mentioned earlier, existing elements of the 5G CN that explicitly handle the control and user plane traffic are augmented with new functionalities, so they could also support multi-access.

AT3S is integrated in the control and user planes thanks to appropriate AT3S extensions in:

User Data Repository (UDR): UDR-AT3SF, which records the UE subscription data related to operator services and user profiles,

Policy Control Function (PCF): PC-AT3SF, which defines and provides the AT3S rule policies to a UE,

Session Management Function (SMF): SM-AT3SF, which provides policies enforcement and AT3S session management. In addition, it create usage reports and receives others from the UE to process them for dynamic AT3S operation,

User Plane Function (UPF): UP AT3SF, composed of a data (control, respectively) plane component Upu-AT3SF (Upc-AT3SF, respectively). UPu-AT3SF is supported by a PDU session anchor representing the UP anchor point for all AT3S traffic and presents a single IP address towards Data Network (DN) via N6. It is responsible for AT3S policy rule enforcement in the User Plane (UP) of the CN. UPc-AT3SF requests, receives and processes traffic usage reports from the UE (if available) as indicated by the SM-AT3SF via N4. It is responsible for setting up the multi-access connectivity through UPu-AT3SF and steering/splitting the traffic over available accesses based on path quality estimation, network access measurement reports and network policies provided by the SM-AT3SF via N4 interface.

UE Access Traffic Steering Switching and Splitting Function (UE-AT3SF): AT3S policy rule enforcement at the UE for UE-initiated traffic. It may also generate traffic reports to be sent to the SM-AT3SF at the request of UPc-AT3SF.

Figure 3-1 Non roaming AT3S architecture - initial high level view [8]

Note that the Access and Mobility management Function (AMF), the Authentication Server Function (AUSF), and the Application Function (AF) are not impacted. Conversely, User Plane Function (UPF), Session Management Function (SMF), Policy Control Function (PCF) and the Unified Data Management (UDM) are impacted by the introduction of the AT3S function. These changed are applied to the cited functions in order to keep track of the different streams, which are split from different flows and distinguish the ones that reassemble the same flow.

SaT5G (761413) D4.3 May 2020

3.2.1 Multi-access Sessions In 5G, like in 4G, UE needs to set up a (data) connection in order to connect to Data Network (DN). In 5G terminology, this is what we call setting up PDU session, and is typically being set up based on the user profile and operator policies. Details can be found in 3GPP TS23.502 [64].

Unlike in 5G PDU session where only one PDU session was set up, in the Multi-Access PDU (MA PDU) session, multiple PDU sessions (at least two PDUs) are running in parallel, which is considered by the rest of the system as a single MA PDU session. Indeed, from the DN, the MA PDU session is seen as a single running session. It is the role of the CN to take care of the multiple PDU sessions running inside a single MA PDU session.

Figure 3-2 taken from the document 3GPP TR 23.793 [8] (please refer to section 6.1.1) depicts a UE accessing a server application via two PDU sessions, which are considered as one MA PDU. In case of AT3S, two accesses have been used, each of which sets up a separate PDU session. The two PDU sessions are sharing two boxes namely, the UPF ( PDU Session Anchor (PSA)) function and the multi-access radio in UE. The former, PSA, which is running within the UPF, splits incoming data stream into two separate PDU sessions, while the latter, multi-access is in UE, joins these two PDU sessions together.

Figure 3-2 MA PDU session

There are several methods to split incoming data stream (coming via N6) using the aforementioned algorithms. In 5G, UE is responsible for establishing new PDU session(s) based on UE Route Selection Policy (URSP) provided by the network (PCF) and/or preconfigured in the UE as described in 3GPP TS 23.503 clause 6.6.2 [65]. URSP provides level steering capability for UE to establish new PDU session(s) over the preferred AN based on traffic descriptor. After the UE had initiated PDU sessions setup, it is up to network to check whether the UE has the rights to use such PDU session (profile, aggregated data rate) and accept or reject that request – cf. Figure 3-3.

With the introduction of MA PDU, it becomes more complicated to set up a connection, as there are more parameters to take into account. Still, user profile, operator’s preferences are main elements determining which accesses are to be used.

3GPP

access

Non-3GPP

accessN3IWF

UE

(including

RG)

UPF

(PSA)

N3

N3

Application

clientServer host

UPF

UPF

N9

N9

MA PDU

PDU

PDU

(linked)

N6

SaT5G (761413) D4.3 May 2020

Traffic Descriptors for

MA PDU

UE-AT3SF

UP-AT3SF

SMF

SM-

AT3SFATSSS configuration for UE

ATSSS configuration for UPF/PSA

ATSSS policy

N4ATSSS rules

PCF

PC-

AT3SFURSP

Control Plane

User Plane

Non-3GPP access

3GPP access

N3IWF UPF

UPF

N3

N3

N9

N9

User preference

settings

Local

configuration

N1 (via AMF)

N1 (via AMF)

UDM

UDR-

AT3SF

N7

N25

N10

UE Policy subscription

UE profiles & subscription for ATSSS

Control Plane

ATSSS rules

User Plane

Measurement reports from UE

Figure 3-3 Interaction between AT3SF's of UE, SMF, PCF & UPF

When a MA PDU session is invoked by the UE, further traffic steering, switching and splitting policy can be provided by the PC-AT3SF to SM-AT3SF during the MA PDU session establishment. Some examples of AT3S configurations provided by SM-AT3SF can be shown as follows:

Load balance: traffic is split over both accesses for equal or weighted distribution,

Hot-standby: traffic is steer to the least cost access, e.g. Wireless Local-Area Network (WLAN) and switch to the other access when the least cost access is no longer available or its link condition falls below certain performance thresholds based on for example, link quality, throughput, latency, and packet loss,

Top up: traffic is steer to least cost access (e.g. WLAN) and if needed split across both accesses when the performance criteria of the least cost access falls below certain performance thresholds based on for example link quality, throughput, latency, and packet loss,

Least loaded: traffic is steer to the least loaded access and switch to the other access when the least loaded access is no longer available or its link condition falls below certain performance thresholds based on for example link quality, throughput, latency, and packet loss,

Best performance: traffic is steer to the best performing access based on traffic measurements and switch to the other access when the best performing access is no longer available.

Based on AT3S configurations provided by the SM-AT3SF and optionally user preference settings, the UE-AT3SF derives the AT3S rules for MA PDU session data in the uplink direction. Similarly, the UP-AT3SF derives the AT3S rules for MA PDU session data in the downlink direction based on AT3S configurations received from SM-AT3SF.

In order to keep optimal split of data into streams, it is possible for UPc-AT3SF to request from UE-AT3SF to provide feedback to PSA on performance of links (including, packet loss and re-transmission), although this is not mandatory. In addition, it is possible to set up thresholds, which when reached will trigger sending of a report to PSA. Examples are latency and packet loss.

3.2.2 User Plane AT3S methods (AT3S Functions) The AT3S architecture is based on Multi-access PDU session. A MA PDU allows application to send and receive traffic either over 3GPP access, or non-3GPP access, or even on both access technologies simultaneously. A MA PDU session comprises of a PDU session over 3GPP access and a "linked" PDU session over non-3GPP access, or vice versa. Each of the PDU sessions may have its own set of UPFs, but both PDU sessions share a common PDU session anchor (PSA).

An exhaustive list of multi-access methods are studied including:

1. Null tunnelling (for traffic steering), 2. Generic Routing Encapsulation (GRE) tunnelling, 3. Generic Multi-Access (GMA) tunnelling, 4. Layer 4 multi-path (MPTCP, MPQUIC,SCTP), UDP generic.

SaT5G (761413) D4.3 May 2020

The above tunnelling methods (2,3,4) encapsulate the IP packets. For instance, in case of AT3S function using the MPTCP convergence method, the payload of the MPTCP subflows are IP packets from or to the UE.

3.2.3 User plane MPTCP method (MPTCP function) The MPTCP function provides multi-path capacities to the TCP traffic. From the UE’s perspective, this is seen as an MPTCP proxy intercepting the TCP connections and transport the TCP payloads using MPTCP, while on the central site, an MPTCP proxy ends the MPTCP operation and restores the original TCP connection towards the internet.

The MPTCP function uses MA PDU sessions but bypass any AT3S function and is negotiated in the same way as an AT3S function. It applies to the entire TCP traffic or to a part of it and can co-exist with any AT3S function. For example, some TCP flows are processed with an MPTCP function while other TCP or UDP flows are processed in tunnel mode by an AT3S function (including, the MPTCP method) and some others are not involved at all in any multi-access operation.

The MPTCP function can use information from the link measurement function in order to perform the path selection decisions (e.g., interactive flows on low delays links, downloads on high bandwidth links).

3.2.4 AT3S limitations In all cases, AT3S with either an AT3S or a MPTCP function, considers only the local UE interfaces, i.e. mainly 5G and WLAN – cf. Figure 3-4. The cases where the satellite links are placed on backhauling equipment (e.g., NTN Terminal and Gateway) and the Backhaul as an hybrid architecture (Satellite terrestrial) are not considered. This reduces the chance to have a satellite link well integrated. Considering this limitation, the following sections will investigate the integration of satellite into a 5G backhaul by building an hybrid backhaul.

Figure 3-4 AT3S using WLAN and 5G interfaces

3.3 Backhaul Traffic Steering, Switching, and Splitting: BT3S It is generally agreed that the backhaul is the best opportunity to integrate satellite communication in the short term.

In cases where the RAN nodes (e.g. HgNodeB) are connected via multiple types of backhaul connections, including (low-bandwidth) DSL/cellular networks and (high-bandwidth) satellite connections, there is a need to have mechanism to handle Backhaul Traffic Steering, Switching, and Splitting (BT3S), also introduced in Sat5G deliverable D3.3. The Steering, Switching, and Splitting policies have been described in Section 3.

In such architecture cases, such as residential gateways, non 5G hosts connected to the gateway can take advantage of the availability of Steering, Switching, and Splitting functions over the multiple available links.

In this section we turn our attention to the Steering, Switching, and Splitting functions and protocols that can be used or adapted in order to build the hybrid (Satellite and Terrestrial) backhauling architecture.

SaT5G (761413) D4.3 May 2020

UEgNB

gNB-BT3SF

N3IWF

AMF

AUSF

UPF

UPc-BT3SF

Upu-BT3SF

Data network

SMF

CP-BT3SF

PCF

PC-BT3SF

UDM

UDR-BT3SF

AF

Figure 3-5 Multi-link in backhaul using BT3S

Figure 3-5 proposes a possible architecture in the case multiple links are used in the backhaul. Here, backhaul specific functions are similar to the AT3S functionalities specified in section 3.2.4. AT3S focuses on access while BT3S addresses the backhaul links between gNB and Core Network. The backhaul could combine 3GPP (5G) and non-3GPP links. In the latter case N3IWF plays a role in order to integrate a non-3GPP link in to the 3GPP architecture.

3.3.1 5G UE using an AT3S or MPTCP function When the 5G UE supports AT3S or MPTCP, a solution is to extend the AT3S in order to have in the AT3S UE the possibility to establish several MA PDU child sessions over the 5G interface, e.g. one child PDU session for the satellite backhaul link, one for the terrestrial backhauling link, and to DSCP mark the packets accordingly – cf. Figure 3-6.

Then the hybrid backhauling gateway performs policy based routing to transmit the corresponding packets on the appropriate links. Another flag may be used if DSCP is modified by the satellite system.

SaT5G (761413) D4.3 May 2020

Figure 3-6 BT3S with AT3SF UE

3.3.2 UE not using an AT3S or MPTCP function When the UE does not support AT3S or MPTCP, a solution is to implement in the hybrid backhauling gateway an MPTCP proxy that intercepts the TCP flows inside the GTP-U traffic and use MPTCP to transport the traffic over the available links – cf. Figure 3-7. The GTP domain is shown at the top of the figure.

At the central site, an MPTCP proxy ends the MPTCP subflows and deliver the original TCP traffic.

Figure 3-7 BT3S with non AT3SF UE

3.3.3 Regular hosts connected to the hybrid backhaul gateway Non 5G hosts must be considered. Provided that the hybrid backhauling gateway acts as an hybrid UE with a satellite and a terrestrial interfaces, a possible solution is to use the regular 5G AT3S function to transport the traffic of these hosts on the available links – cf. Figure 3-8.

SaT5G (761413) D4.3 May 2020

Figure 3-8 BT3S with regular (non 5G) hosts

3.4 Mapping AT3S to SaT5G architecture options In integrated satellite-terrestrial network, the different backhaul (or indirect UE access) implementation options were already detailed in SaT5G deliverable D3.1 and in ETSI [66]. Two major variants have emerged: transport network based implementation options, and relay node based implementation options. For both variants, two sub-cases can be distinguished according to the satellite access either it is based on 3GPP defined protocols, or not. This approach leads to five different possible variants that are illustrated herein. Relying on Sat5G General Network Architecture D3.1, the general concepts regarding the transport network were already provided. For the sake of providing a self-contained output, the two options are illustrated and briefly discussed.

Figure 3-9 Backhaul implementation options identified by SaT5G

As discussed in SaT5G “Business Modelling and Techno-economic Analysis of Satellite eMBB” D2.3, it is proposed that each Satellite Network Operator (SNO) runs a component called a Resource

SaT5G (761413) D4.3 May 2020

Manager (RM) function, which keeps track of all available resources, collects usage, and makes decisions. The RM function can be either centralized or decentralized. If the management functions and decision-making used to allocate resources are concentrated at a central component, there can be problems of lack of scalability when the system size increases.

Sat5G General Network Architecture D3.1 shows two main options for use of satellites – as a backhaul and for direct access. The same options are still valid when we look at the multi-link access. For multi-access, we will start from 3GPP architecture, and functional division in blocks of the CNs, as shown in Figure 3-9. Therefore, since ‘starting point’; of the splitting / switching is already defined, our interest lies in ‘termination point’ of this split. That termination point can be located in UE (as in direct access) or in eNB (backhaul option).

SaT5G (761413) D4.3 May 2020

4 Satcom QOS in to 5G

Efficient management of available resources in 5G networks is of paramount importance and WP3 links 5G QI and Satcom CoS. Optimizing available resources to better match the requirements in a dynamic changing environment is a challenge in SaT5G systems. In Sat5G D2.3 and D3.2 Integrated SaT5G Detail Network Architecture, it is proposed that Satellite Network Operators (SNOs) deploy and exploit a Resource Manager (RM) function to handle resource allocation and orchestration. A Decentralized Resource Manager (DRM) approach distributes the management functions and decision-making units over different network management components. The DRM approach facilitates scalability of the network and avoids possible congestion in the case of a Centralized Resource Manager.

4.1 Connection setup When a UE is activated, several actions take place. Two phases, as described in Section 3.2.2 in D3.1 can be distinguished:

Phase 1 in which Connection, Registration, Mobility Management, Authentication and Slice Selection processes take place;

Phase 2 in which Session Management, User Plane Management, Policy Control and Quality of Service handling occurs.

Phase 1: Initially, the RAN nodes (e.g. gNodeBs) establish a connection to the 5G CN over the N2 interface. When being activated a UE initiates a Radio Resource Control (RRC) connection to RAN nodes with NAS signalling towards an AMF and provides a list of slice identities for this connection. The RAN node selects the specified AMF and forwards the Non-access Stratum (NAS) signalling to it. In this fashion, NAS signalling is being established over the N1 interface. Subsequently, the Network Slice Selection Function (NSSF) is queried for the relationship between the functions in the slice specified by the UE. At this stage, the UE is authenticated through Authentication Server Function (AUSF) and Unified Data Management (UDM). An SMF is selected that handles the upcoming PDU session of the UE. The above listed steps make part of the Registration and Mobility Management procedure by which the UE becomes registered in the 5G Core Network and finds itself in the Connected mode.

Phase 2: After being registered, the UE may initiate one or multiple PDU sessions. These sessions are established by the SMF which creates UP connections through one or more User Plane Functions (UPFs) and applies Policy Control through the Policy Control function (PCF) which establishes the appropriate Quality of Service (QoS) for the PDU Sessions.

4.2 QoS in 5G The QoS aspects in 5G are specified in 3GPP TS 23.501 [67]. These QoS aspects apply to terrestrial 5G connections as well as to the satellite links that comply to the 5G standard (i.e., direct connectivity). The 5G QoS model is based on QoS Flows, which supports the following:

Guaranteed Bit Rate (GBR) QoS Flows,

Non-GBR QoS Flow.

5G QoS Flow represents the finest granularity for QoS differentiation in the PDU session of the 5G System. All traffic mapped to the same 5G QoS flow receives the same forwarding treatment (including, scheduling policy, queue management policy, rate shaping policy, and Radio Link Control (RLC) configuration). Providing different QoS forwarding treatment requires separate 5G QoS Flow. In the 5G systems, a QoS flow is controlled by the SMF.

5G QoS Flow Identifier (QFI): is a scalar used to identify a QoS flow in the 5G system. User plane traffic with the same QFI within a PDU Session receives the same traffic forwarding treatment (e.g. scheduling and admission threshold). The QFI is carried in an encapsulation header on N3 (and N9). The QFI shall be unique within a PDU Session. The QFI may be dynamically assigned or may be equal to a 5G QoS Identifier (5QI) - a scalar that is used as a reference to 5G QoS characteristics (see

Table 1 for a set of standardized 5QI values).

Any QoS Flow is characterized by the following, as defined in the document 3GPP TS 23.501 [67]:

A QoS profile provided by the SMF to the AN via the AMF over the N2 reference point or preconfigured in the AN,

SaT5G (761413) D4.3 May 2020

One or more QoS rule(s) which can be provided by the SMF to the UE via the AMF over the N1 reference point and/or derived by the UE by applying reflective QoS control,

One or more Service Data Flow (SDF) template(s) provided by the SMF to the UPF.

4.2.1 QoS Profiles Each QoS flow is characterized by a corresponding QoS profile detailed in Table 1 [67]. Distinctions are made among QoS profiles related to GBR or Non-GBR, namely:

a) GBR QoS profile is defined by the following parameters: o 5G QoS Identifier (5QI),, o Allocation and Retention Priority (ARP)- contains information about the priority level, the

pre-emption capability and the pre-emption vulnerability. The ARP priority level defines the relative importance of a resource request. This allows deciding whether a new QoS flow may be accepted or needs to be rejected in the case of resource limitations (typically used for admission control of GBR traffic). It may also be used to decide which existing QoS flow to pre-empt during resource limitations. The range of the ARP priority level is 1 to 15 with 1 as the highest level of priority,

o Guaranteed Flow Bit Rate (GFBR) – Uplink (UL) and Downlink (DL) denotes the bit rate that is guaranteed to be provided by the network to the QoS Flow over the Averaging Time Window (time interval over which the GFBR and MFBR are calculated). Bit rates above the GFBR value and up to the MFBR value, may be provided with relative priority determined by the Priority Level of the QoS Flows,

o Maximum Flow Bit Rate (MFBR) - UL and DL limits the bit rate to the highest bit rate that is expected by the QoS Flow (e.g. excess traffic may get discarded or delayed by a rate shaping or policing function at the UE, RAN, UPF),

o Notification control - indicates whether notifications are requested from the NG-RAN when the GFBR can no longer (or can again) be guaranteed for a QoS Flow during the lifetime of the QoS Flow. Notification control may be used for a GBR QoS Flow if the application traffic is able to adapt to the change in the QoS (e.g. if the AF is capable to trigger rate adaptation),

o Maximum Packet Loss Rate - UL and DL indicates the maximum rate for lost packets of the QoS flow that can be tolerated in the uplink and downlink direction.

b) Non-GBR QoS Profile is defined by the following parameters: o 5QI, o ARP, o The Reflective QoS Attribute (RQA) is an optional parameter which indicates that certain

traffic (not necessarily all) carried on this QoS Flow is subject to Reflective QoS. Reflective QoS enables the UE to map UL User Plane traffic to QoS Flows without SMF provided QoS rules and it applies for IP PDU Session and Ethernet PDU Session. This is achieved by creating UE derived QoS rules in the UE based on the received DL traffic. It shall be possible to apply Reflective QoS and non-Reflective QoS concurrently within the same PDU Session.

Table 1 provides an overview of 3GPP standardized 5QI values.

Table 1 Standardized 5QI to QoS characteristics mapping [67]

SaT5G (761413) D4.3 May 2020

5QI Value

Resource Type

Default Priority Level

Packet Delay

Budget

Packet Error Rate

Default Maximum Data Burst

Volume (NOTE 2)

Default Averaging Window

Example Services

1

GBR

(NOTE 1)

20 100 ms

(NOTE 11, NOTE 13)

10-2 N/A 2000 ms Conversational Voice

2

40 150 ms

(NOTE 11, NOTE 13)

10-3 N/A 2000 ms Conversational Video (Live Streaming)

3 (NOTE 14)

30 50 ms

(NOTE 11, NOTE 13)

10-3 N/A 2000 ms

Real Time Gaming, V2X messages Electricity distribution – medium voltage, Process automation - monitoring

4

50 300 ms

(NOTE 11, NOTE 13)

10-6 N/A 2000 ms Non-Conversational Video (Buffered Streaming)

65 (NOTE 9, NOTE 12)

7 75 ms

(NOTE 7, NOTE 8)

10-2 N/A 2000 ms Mission Critical user plane Push To Talk voice (e.g., MCPTT)

66 (NOTE 12)

20

100 ms (NOTE 10, NOTE 13)

10-2 N/A 2000 ms Non-Mission-Critical user plane Push To Talk voice

67 (NOTE 12)

15 100 ms

(NOTE 10, NOTE 13)

10-3 N/A 2000 ms Mission Critical Video user plane

75 (NOTE 14)

5

Non-GBR (NOTE 1)

10 100 ms

NOTE 10, NOTE 13)

10-6 N/A N/A IMS Signaling

6 60 300 ms

(NOTE 10, NOTE 13)

10-6 N/A N/A

Video (Buffered Streaming) TCP-based (e.g., www, e-mail, chat, ftp, p2p file sharing, progressive video, etc.)

7 70 100 ms

(NOTE 10, NOTE 13)

10-3 N/A N/A

Voice, Video (Live Streaming) Interactive Gaming

8 80 300 ms

(NOTE 13) 10-6 N/A N/A

Video (Buffered Streaming) TCP-based (e.g., www, e-mail, chat, ftp, p2p file sharing, progressive

9 90 video, etc.)

69 (NOTE 9, NOTE 12)

5 60 ms

(NOTE 7, NOTE 8)

10-6 N/A N/A

Mission Critical delay sensitive signaling (e.g., MC-PTT signaling)

70 (NOTE 12)

55 200 ms

(NOTE 7, NOTE 10)

10-6 N/A N/A

Mission Critical Data (e.g. example services are the same as 5QI 6/8/9)

SaT5G (761413) D4.3 May 2020

5QI Value

Resource Type

Default Priority Level

Packet Delay

Budget

Packet Error Rate

Default Maximum Data Burst

Volume (NOTE 2)

Default Averaging Window

Example Services

79 65 50 ms

(NOTE 10, NOTE 13)

10-2 N/A N/A V2X messages

80 68 10 ms

(NOTE 5, NOTE 10)

10-6 N/A N/A Low Latency eMBB applications Augmented Reality

82

Delay Critical GBR

19 10 ms

(NOTE 4) 10-4 255 bytes 2000 ms

Discrete Automation (see TS 22.261 [2])

83 22 10 ms

(NOTE 4) 10-4

1354 bytes (NOTE 3)

2000 ms Discrete Automation (see TS 22.261 [2])

84 24 30 ms

(NOTE 6) 10-5

1354 bytes (NOTE 3)

2000 ms Intelligent transport systems (see TS 22.261 [2])

85 21 5 ms

(NOTE 5) 10-5 255 bytes 2000 ms

Electricity Distribution- high voltage (see TS 22.261 [2])

NOTE 1: A packet which is delayed more than PDB is not counted as lost, thus not included in the PER. NOTE 2: It is required that default MDBV is supported by a PLMN supporting the related 5QIs. NOTE 3: This MDBV value is set to 1354 bytes to avoid IP fragmentation for the IPv6 based, IPSec protected

GTP tunnel to the 5G-AN node (the value is calculated as in Annex C of TS 23.060 [56] and further reduced by 4 bytes to allow for the usage of a GTP-U extension header).

NOTE 4: A delay of 1 ms for the delay between a UPF terminating N6 and a 5G-AN should be subtracted from a given PDB to derive the packet delay budget that applies to the radio interface.

NOTE 5: A delay of 2 ms for the delay between a UPF terminating N6 and a 5G-AN should be subtracted from a given PDB to derive the packet delay budget that applies to the radio interface.

NOTE 6: A delay of 5 ms for the delay between a UPF terminating N6 and a 5G-AN should be subtracted from a given PDB to derive the packet delay budget that applies to the radio interface.

NOTE 7: For Mission Critical services, it may be assumed that the UPF terminating N6 is located "close" to the 5G_AN (roughly 10 ms) and is not normally used in a long distance, home routed roaming situation. Hence delay of 10 ms for the delay between a UPF terminating N6 and a 5G_AN should be subtracted from this PDB to derive the packet delay budget that applies to the radio interface.

NOTE 8: In both RRC Idle and RRC Connected mode, the PDB requirement for these 5QIs can be relaxed (but not to a value greater than 320 ms) for the first packet(s) in a downlink data or signaling burst in order to permit reasonable battery saving (DRX) techniques.

NOTE 9: It is expected that 5QI-65 and 5QI-69 are used together to provide Mission Critical Push to Talk service (e.g., 5QI-5 is not used for signaling). It is expected that the amount of traffic per UE will be similar or less compared to the IMS signaling.

NOTE 10: In both RRC Idle and RRC Connected mode, the PDB requirement for these 5QIs can be relaxed for the first packet(s) in a downlink data or signaling burst in order to permit battery saving (DRX) techniques.

NOTE 11: In RRC Idle mode, the PDB requirement for these 5QIs can be relaxed for the first packet(s) in a downlink data or signaling burst in order to permit battery saving (DRX) techniques.

NOTE 12: This 5QI value can only be assigned upon request from the network side. The UE and any application running on the UE is not allowed to request this 5QI value.

NOTE 13: A delay of 20 ms for the delay between a UPF terminating N6 and a 5G-AN should be subtracted from a given PDB to derive the packet delay budget that applies to the radio interface.

NOTE 14: This 5QI is not supported as it is only used for transmission of V2X messages over MBMS bearers as defined in TS 23.285 [72].

4.2.2 QoS Rules The 3GPP TS 23.501 [67] specifies how the UE performs the association of UL User plane to QoS Flows based on QoS rules. These QoS rules may be explicitly provided to the UE (i.e., explicitly signalled QoS rules using the PDU Session Establishment/Modification procedure), pre-configured in the UE or implicitly derived by the UE by applying Reflective QoS. A QoS rule contains:

The QFI of the associated QoS Flow,

A Packet Filter Set used to identify one or more packet flows and,

SaT5G (761413) D4.3 May 2020

A QoS rule precedence value and the Packet Detection Rule (PDR) precedence value determine the order in which a QoS rule or PDR shall be evaluated. The evaluation of the QoS rules or PDRs is performed in increasing order of their precedence value.

An explicitly signalled QoS rule contains a QoS rule identifier which is unique within the PDU Session and is generated by SMF.

Figure 4-1 illustrates the classification, marking and mapping of the User Plane traffic to Access Network (AN) resources. As elaborated in D3.2, QoS mapping is distinguished in two directions, namely:

In download, data packets are classified by PDR in UPF using Service Data Flow (SDF) templates to generate QoS flows which are further mapped to AN resources. The SMF performs the binding of SDFs to QoS Flows based on the QoS and service requirements (as defined in TS 23.503),

In upload, data packets are mapped to QoS Flows and those are marked using QoS Rules. Subsequently these QoS Flows are mapped to AN resources.

4.2.3 Legacy Satellite QoS (Class of Service) The QoS principle used in satellite communications is described in section 3.3.1 of D3.2 and is summarized in this section. A mechanism called Differentiated Services (DS) can be used as a possible QoS classification over satellite networks. Differentiated services or DiffServ is a computer networking architecture that specifies a simple and scalable mechanism for classifying and managing network traffic and providing quality of service (QoS) on IP networks. DiffServ uses a 6-bit differentiated services code point (DSCP) in the 8-bit differentiated services field (DS field) in the IP header for packet classification purposes.

D3.2 states that VSAT networks often support multiple virtual networks; each within their own bandwidth pool. These tools require the VSAT hub and terminal having sufficient visibility to be able to identify the packet and send it to the required queue. If the data is encrypted within an IPsec tunnel then either; (a) a copy of the keys is provided to allow the system visibility to identify the packet priority (i.e., sufficient trust is provided to have the right keys for this purpose; (b) the DCSP flag is set correctly; or (c) the data is sent at the default priority. Packets sent using either unencrypted tunnelling or encryption can be better identified.

Some VSATs can switch return channel mode to switch between optimum support for normal bursty traffic, and to support high bandwidth and/or low jitter applications. The rules are defined on the VSAT

AN UPF UE

Data packets from applications

QoS rules (mapping UL packets to QoS flows

and apply QoS flow marking)

Mapping QoS flows to AN

Resources

QoS Flow (all packets marked with

the same QFI)

PDU Session

PDRs

(classify packets for QoS flow marking and other actions)

Application /Service Layer

AN Resources

Figure 4-1 Classification principle, User Plane QoS Flows marking and AN Resources mapping

SaT5G (761413) D4.3 May 2020

network management system and implemented in real time by: (a) a network function such as a network control centre in the VSAT hub for traffic in the forward link, and (b) by the VSAT in coordination with the network control centre. Typically these VSAT systems support four priority levels of IP based traffic, with a fifth higher level offering constant bit-rate. Some vendors implement IP multicast traffic in one of the IP traffic levels, others create an additional level for multicast only.

Notes: (1) Satellite links generally operate in quasi-error-free mode for unicast traffic, therefore the service class selection can have little relevance. (2) Latency on a correctly dimensioned satellite system tends to be somewhat fixed (the satellite path length dominating, with queuing delays secondary), however higher priority packets will have smaller queuing delays unless that higher priority bandwidth pool is overloaded. Further on D3.2 provides a list of requirements for evolution to facilitate 5G terrestrial-satellite interworking – cf. Table 2 as given in D3.2Table 2 An example of mapping between satellite CoS and 5QI and Figure 4-2:

Standards based key management to allow access to IPsec keys where sufficient trust exists,

Process to accept slice request with 5G QI enabling a feedback mechanisms (e.g. to state the requested criterion is not met),

Clear standards based mapping between satellite CoS and 5G classes (5G QI); An example of possible mapping is given in Table 2 and illustrated in Figure 4-2 for the case where satellite connectivity is used as transport network (TN) for backhauling as in D3.2.

Table 2 An example of mapping between satellite CoS and 5QI

SaT5G (761413) D4.3 May 2020

4.3 QoS in SaT5G hybrid backhaul Currently, the way in which the SMF becomes and stays aware of the QoS properties of the backhaul is not considered in the 3GPP standards. Backhaul is assumed to be arranged by network operator, and 3GPP is considering it out of scope. On the other hand, in case that we use satellites or hybrid access as a backhaul, QoS conditions in the backhaul will be changing, and therefore it is important to study a solution for this QoS Backhaul Awareness.

This section analyses QoS aspects in the case of hybrid (satellite and terrestrial) backhaul connectivity. We consider mechanism which is similar to one used in hybrid access – AT3S (as described in Section 3), and since one we consider is focused on backhaul we will call it BT3S (Backhaul Traffic Splitting, Steering and Switching). We will look here at methods to monitor QoS of multipath links with goals to select one of the BT3S modes of operation. In addition, required adaptation of 3GPP architecture is proposed in order to support QoS control (with focus on latency) over the multipath connectivity as well as BT3S.

Note that 3GPP as of this moment does not have explicitly defined BT3S function, but that it does allow for hybrid backhaul.

Hybrid backhaul is used between the Core Network and the radio access node and combines two types of transport networks e.g. terrestrial and satellite or even a combination of two satellite transport layers (MEO and GEO). Without the loss of generality we will further focus the presentation on the former case.

The solution is based on a combination of architectures and procedures described in AT3S (3GPP TR 23.793) and in ‘Solution#11: Backhaul QoS handling based on AMF and UPF information’ (specified in 3GPP TR 23.737). The following assumptions apply:

Satellite and terrestrial links are combined in a backhaul link (N3 interface between 5GC (core network) and gNB (Access Network),

AMF is aware of the types of the links combined in the hybrid backhaul,

As the satellite link used for backhauling is considered to be non-3GPP technology, the 5G system should be capable of converting the Sat Classes of Service (CoS) to the 5QI parameters used in 3GPP,

All AT3S functions specified in Section 3.1, 3.2 and 3.4 are replaced by (corresponding) BT3S functions. A novel function gNB-BT3S is introduced at AN Node (gNB) to support the

Figure 4-2 Adaptation of QoS configurations between satellite and terrestrial networks

SaT5G (761413) D4.3 May 2020

backhaul Steering, Switching and Splitting (SSS). Accordingly, a UE does not require SSS functionality,

Control plane data may be sent over both transport links of the backhaul. However, a link with lower latency is always given the priority in transferring control data.

A possible system architecture for the hybrid backhaul proposes the use of backhaul Traffic Steering Switching and Splitting functions. The hybrid backhaul is established between gNB-BT3SF and UPF that acts as the PDU Session Anchor – just like it did in AT3S.

The usage of satellite links could result in QoS limitations of the provided services due to larger latency compared to terrestrial links. To properly handle such cases, AMF and UPF (and their corresponding BT3S functions) should be aware of backhaul link properties and their characteristics (e.g. latency). Those can be determined by the AMF configuration or via signalling between AN (gNB) and 5GC for each of the transport links. For the case of UPF this awareness is either pre-provisioned or dynamically determined by monitoring QoS properties of the connections.

The information on backhaul links’ QoS available at AMF and/or UPF is subsequently transferred to SMF and PCF. The knowledge of QoS limitations in each of the backhaul links can be used as follows,

The SMF can use the QoS limitation of the links for the dynamic selection of one of the 3 possible SSS modes in the UPF and AN (gNB),

After UPF selection, the SMF can inform the PCF via an SM Policy Association Modification procedure about the inability to apply the provided PCC. The PCF can use this information (possibly after informing/consulting an AF or based on the service characteristics provided by the AF when requesting QoS over N5) to adjust the QoS parameters in the PCC rule provided to the SMF or to request the SMF to release the PDU Session in case requested PCC rules are impossible to be achieved.

SaT5G (761413) D4.3 May 2020

5 5G multi-link prototypes

A subset of components of the above ideal solution is implemented and demonstrated on a local Ekinops Amarisoft 4G testbed and on University of Surrey’s 5GIC testbed. The MPTCP prototype from VITAL has been enhanced to support 4G/5G GTP encapsulation. The Amarisoft testbed allows to validate on a well-known reference 4G Core Network while the 5GIC deployment proves the prototypes is compatible and works in Surrey’s University near real-world 5G Core Network.

In the scope of the SAT5G project we implement and test the BTSSS concept using MPTCP and MPQUIC:

MPTCP is implemented by a MPTCP proxy on an hybrid terminal at the remote and an hybrid gateway at the central. The hybrid equipment provide an hybrid backhauling network including a satellite link. The proxy intercepts TCP connections from non AT3S UE, and hosts connected to an UE or to the hybrid terminal itself and use MPTCP on the hybrid network,

MPQUIC is implemented on a Client connected to an UE and a Server at the central. Both client and server operate MPQUIC establishing 2 subflows over the 5G network and mark the traffic of each sub flow. Policy based routing occurs in the hybrid terminal and gateway in order to route each sub flow on the appropriate link.

For MPTCP and MPQUIC path selection based on objects length is implemented in addition to Weighted Round Robin.

The MPTCP and MPQUIC test plans aim at measuring the User Experience. The test plans are different because the underlying TCP and UDP protocols do not support the same applications. And comparing MPTCP and MPQUIC is not among the targets of this project.

5.1 MPTCP multi-link prototype In order to perform MPTCP multi-link operations, we introduce two intermediate network functions in the backhaul network namely, the hybrid terminal function and the hybrid gateway function. Besides being able to intercept end users TCP traffic in GTP-u encapsulated packets, these functions run several software modules necessary for link selection, traffic switching and splitting, e.g. MPTCP proxy, Link status and bandwidth monitoring modules. The MPTCP proxies located on both sides of the backhaul network turn incoming TCP flows into MPTCP subflows and then invoke the link selection module that takes care of choosing a path for each segment based on the path selection algorithm in use.

To summarize, all the software modules implemented in our two network functions collaborate together in order to achieve the following actions:

Detect GTP header and store metadata in memory as depicted below in Figure 5-1,

Figure 5-1 GTP header detection

Remove GTP header once the two TEIDs (GTP tunnel endpoints eNodeB->EPC and EPC->eNodeB) are stored,

Intercept TCP flows in the de-capsulated user traffic and establish MPTCP subflows depending on the available links status and the path selection algorithm,

Encrypt user traffic (I-e inner IP) using IPsec ESP,

Restore the original GTP header (that was stored in memory),

Send GTP-u encapsulated MPTCP segments over the chosen links, the layout of the packets over the two backhaul links is depicted in Figure 5-2,

SaT5G (761413) D4.3 May 2020

Figure 5-2 Packet layout over the backhaul links

Decrypt the incoming traffic over the backhaul links and forward it via the local interface (I-e. towards eNodeB or EPC).

As depicted in Figure 5-3, choosing a good path selection algorithm is the key to having significant bandwidth aggregation and noticeable latency reduction. Later in this section we compare some path selection algorithms and show that PSBOL+OFFLOAD outperforms all of them.

The experimentation initially performed for the H2020 VITAL project is reproduced on Ekinops Amarisoft LTE testbed and on Surrey’s University 5GIC network

On such network, we expect to obtain 18Mbps/50ms equivalent link when combining a satellite 15Mbps/650msec with a cellular 3Mbps/50msec link.

Figure 5-3 Bandwidth aggregation and latency reduction

5.1.1 Setup The Ekinops setup is composed of the following components: an Amarisoft LTE testbed (EPC and SDR-based eNodeB), the hybrid Terminal also known as IUG (Intelligent User Gateway), the hybrid gateway or ING (Intelligent Network gateway), a GEO satellite link emulated by OpenSAND (15Mbps with 650ms RTT), a terrestrial link (3Mbps with 50ms RTT), a user equipment and an HTTP server. The bandwidths have been chosen with a ratio of 5 to be significantly different in order to give easy to interpret and produce results. Higher bandwidth values with the same ratio would give the same results since no system component, except the links, is under stress.

Figure 5-4 depicts the entire architecture of the testbed.

SaT5G (761413) D4.3 May 2020

Figure 5-4 MPTCP prototype testbed

In our lab, we deployed our two intermediate network functions (i.e. ING and IUG) as VNFs using Qemu/KVM as the virtualization platform and Linux bridges for creating necessary virtual networks.

Figure 5-5 and Figure 5-6 ING virtual and physical interfaces depict the deployment architecture of the two VMs on Qemu/KVM, the necessary virtual networks and physical interfaces.

Figure 5-5 IUG virtual and physical interfaces

Figure 5-6 ING virtual and physical interfaces

SaT5G (761413) D4.3 May 2020

As illustrated by Figure 5-5, the VNFs have four virtual interfaces connected to four physical interfaces via Linux bridges. The exact configuration of the virtual interfaces as well as the interface model are described in libvirt trace below:

<?xml version="1.0" ?>

<domain type="kvm" xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0">

<name>ING-eki</name>

….

</interface>

<interface type="bridge">

<source bridge='br_adsl'/>

<target dev="2.multCent"/>

<mac address="52:54:00:02:1c:86"/>

<address bus="0x01" domain="0x0000" function="0x0" slot="0x03" type="pci"/>

<model type="e1000"/>

<mtu size='1500'/>

</interface>

<interface type="bridge">

<source bridge='br_sat'/>

<target dev="3.multCent"/>

<mac address="52:54:00:02:1c:87"/>

<address bus="0x01" domain="0x0000" function="0x0" slot="0x04" type="pci"/>

<model type="e1000"/>

<mtu size='1500'/>

</interface>

Note that the management interfaces on the two VMs are necessary in order to configure them via NETCONF protocol. The Netconf configuration files include multilink routing settings as well as GTP and IPSec tunnel management and configuration.

We use this testbed to run several MPTCP test scenarios using the three categories of TCP traffic described earlier (interactive, download, and mixed) with different path selection algorithms (WRR, Offload, PSBOL, PSBOL combined with Offload). For each scenario we report the expected results as well as the achieved performance.

5.1.2 Results As mentioned in [2.1.1], we consider TCP flows having the following behaviours:

Interactive: exchange of short objects,

Download/upload: exchange of long objects at least in one direction,

Mixed: sometimes interactive, sometimes download/upload.

We emphasis on the fact that most applications follow the mixed behaviour. For instance, an SCP transfer usually considered as a download application, is in fact interactive during the TCP connection establishment and the authentication handshake where short messages are exchanged in both directions. The expected and the real test results are shown in Table 3 and Table 4.

SaT5G (761413) D4.3 May 2020

Table 3 MPTCP expected performance

Algorithm Flow behaviour

May change during the connection lifecycle Interactive Download Mixed

WRR --- ++++ +- OFFLOAD --- ++++ +- PSBOL ++++ ++ ++ PSBOL+OFFLOAD ++++ ++++ ++++ For interactive operations, PSBOL and PSBOL+ Offload are expected to be optimal because short objects are sent on the terrestrial link (lowest RTT link). For these interactive operations, WRR and Offload are expected to provide bad performance because some packets belonging to short objects will be sent over the highest RTT link, increasing the total object load duration. For download/upload operations, PSBOL is expected to be sub-optimal since it doesn’t provide bandwidth aggregation, unlike the other algorithms. For mixed flows, PSBOL+ Offload is expected to provide optimal QOE since both interactive and download operations are fully optimized.

Table 4 PSBOL+ Offload experimental results

Backhaul Link SSH execution time (sec) Traffic Interactive Download Mixed Terrestrial 3Mbps/50ms 5 17 21 Satellite 15Mbps/650ms 48 16* 58 Satellite + Terrestrial PSBOL+ Offload 6 9** 14

Note *: TCP and ssh handshakes over satellite take approximately 6 seconds.

Note **: handshake messages are short and therefore sent over the terrestrial link.

As shown in Table 3 and Table 4, the PSBOL+ Offload algorithm outperforms the other path selection algorithms and improves significantly the QoE across all scenarios. This algorithm cancels the satellite latency negative impact on the QOE and takes advantage of the cumulated satellite and terrestrial links bandwidths. Combining a 15Mbps/650ms Satellite link with a 3Mbps/50ms ADSL link looks like operating a 18Mbs/50ms link.

5.2 MPQUIC multi-link prototype

5.2.1 Setup

On Figure 5-7Figure 5-7 MPQUIC testbed, the following elements are visible:

Remote Multipath done in a PC client connected to a 5G UE,

Central Multipath is done in a PC Server,

Hybrid Term and GW: Linux PC routers with 3 links :Lan Satellite Terrestrial.

Yellow/green elements are multi-link specific.

MP QUIC with Path Selection based on Objects Lengths is used: objects are detected on QUIC streams. 2 MP QUIC subflows are used and DiffServ tagged: Satellite (long objects) and Terrestrial (short objects).

The satellite traffic (N3) is Policy Base Routed by the hybrid Term and Gateway. The demonstration is performed on a Linux PC with all the components required such as Name Spaces, at Ekinops Sophia-Antipolis premises with a 4G Amarisoft network and the satellite emulated, and at Surrey University integrated in the 5GIC 4G/5G testbed.

SaT5G (761413) D4.3 May 2020

Figure 5-7 MPQUIC testbed

5.2.2 Software The QUIC/MPQUIC software is based on QUIC Go and include a Client and a Server that support the HTTP2 protocol over TCP (H2/TCP), over QUIC (H2/QUIC) and over MPQUIC (H2/MPQUIC). The MPQUIC implementation is the prototype proposed by UC Louvain.

The Client operates either in H2/TCP (TCP) or H2/QUIC (default) or H2/MPQUIC (-m) and allows to request several URLs in parallel, e.g. Client https://server1/Obj1 https://server2/Obj2 …. https://serverj/Objn. When all the servers differ then the corresponding objects are loaded in parallel each on a separate connection to the given server (TCP or QUIC or MPQUIC). When two or more URLs refer to the same server then the corresponding objects are sent in parallel on the same connection to the server (TCP or QUIC or MPQUIC) but in different streams.

The Client has been extended to support the following syntax: Client {-tcp} -{m} https://server1://obj1 count, mode … https://serverj/objn,countn,moden . Where Count is the number of iterations and mode is sequential or parallel.

The server supports H2/TCP H2/QUIC and H2/MPQUIC. The Path Selection Algorithm provided in the UC Louvain MPQUIC implementation is Weighted Round Robin(WRR). It is well known that for transmitting long objects, links with higher bandwidth is preferable. Similarly, the transmission of short objects is more efficient on lower delays links. The WRR scheduler is well suited for transmitting long objects since this algorithm provides bandwidth aggregation. However, for short objects ,WRR is under optimal when the delays of the various links are different. In this case some packets of the object are sent on high delay links, which may mean reordering, and the total object transmission delay is always greater than the highest delay

The MPQUIC path selection algorithm on the server has been extended to consider the size of the delivered object in the following way: short objects are sent on the lowest delay link while long objects are sent in WRR mode on all the links.

5.2.3 Test Method The test method consist in executing a set of load cases in various network topologies, using on each topology various application protocols

The following load cases are considered that allow the client to retrieve from the server. Each of case sends objects in parallel, across multiple sessions, or sequentially

a 40KB object 8 times, i.e. 40KBx8, in sequential mode

a 40KB object 8 times in parallel mode

a 286KB object 8 times in sequential mode

a 286KB object 8 times in parallel mode

a 4MB object ,8 times in sequential mode

SaT5G (761413) D4.3 May 2020

a 4MB object ,8 times in parallel mode

a 64MB ,3 times in sequential mode

a 64MB ,3 times in parallel mode

Mix1: 10KBx28 + 15KBx6 + 250KBx2+ 400KBx3+ 700KBx2 in parallel

Mix2: 1.5KBx46 + 5KBx15+ 15KBx32+ 250KBx2+ 400KBx3+ 700KBx1 in parallel.

Mix1 and Mix2 are representative of sites among the top ten Web sites home pages and are characterized by a great number of short and medium objects and the lack of large objects.

The following network topologies and application protocols are tested on links with 5 to 1 bandwidths ratio and relatively low to be easy to produce and interpret:

Using a satellite link alone (50Mbps bandwidth, 600ms RTT) with HTTP2/TCP, then HTTP2/QUIC,

Using an ADSL link alone (10Mbps bandwidth, 20ms RTT) with HTTP2/TCP, then HTTP2/QUIC,

Using the 2 links above together, with HTTP2/MPQUIC using a path selection algorithm WRR scheduler, then PSBOL + WRR: sending short objects on ADSL and long objects on the 2 links in WRR.

5.2.4 Test Results The obtained results for the various test cases are summarized in Table 5. The values represent the total elapsed times in seconds to achieve the download. OLT is Object Load Time and PLT is Page Load Time, a page containing multiple objects. The times include the session(s) setup, the lower values are optimal, in Green. Average values are in Yellow and Red are the worse ones. These times are representative of the user Quality of Experience.

We propose first to analyse globally the Table 5 results and then to consider separately the results for sequential and parallel modes.

Globally, the performances observed on the ADSL link are better than the ones obtained via the satellite link for short objects (cases 1, 2, 3 and 4 as well as for the mixed traffic Mix1 and Mix2). On satellite link, the results are very bad and not in line with the 5G delays expectations. In multi-link operations PSBOL combined with WRR scheduler provides the same optimum performance level as ADSL alone while WRR provide bad results in sequential mode. All these results are in line with the expectations and confirm that short objects are well suited on low delay links. It is also demonstrated that in multi-link operation, optimal performance is achieved when using the path selection algorithm that considers the size of the transmitted objects in order to send all the packets of a short object on the lowest RTT link.

For cases where long objects are involved (cases 5 to 8), better performances are observed, as expected, on high bandwidth links even if the delays are high. In multi-link operations PSBOL combined with WRR, and WRR provide the same optimal performance, which is better than the one provided by each link used alone. Again, these results are in line with the expectations and confirm that long objects are well suited on high bandwidth links. It is also demonstrated that in multi-link operation, optimal performance is achieved when using the path selection algorithm that considers the size of the transmitted object in order to send all the packets of a long object on each of the links in WRR mode.

In Sequential mode, the page load time of a page of N objects is, as expected, nearly N*Object Load Time because the objects are loaded one after one introducing one RTT delay between objects.

In parallel mode, object load times and pages load times are in the same order. This is due to the parallel nature of the request sending and the transfers. This is verified for all the topologies, all the protocols and all the sizes

Pages load times in sequential mode are higher than in parallel mode. This is due to the fact that, in parallel mode, all the objects being also requested in parallel, only one RTT instead of N is wasted to request the objects. This may be also due to congestions at stream level. In sequential mode, there is almost a stream used by connection. If a congestion occurs at stream level, no other object can use the

SaT5G (761413) D4.3 May 2020

bandwidth available and authorized by congestion control at connection level. In parallel mode, several objects are sent in parallel on several streams. If a stream becomes congested, other non-congested streams can benefit from the available bandwidth.

5.2.4.1 Sequential mode The experimental results for the sequential traffic mode are in Table 6 Sequential Mode experimental results. In this table are reported the ratios between the related Page Load Time in Table 5 and the MPQUIC PSBOL Page Load Time in the same table.

For example, the Sat/H2-TCP value: 14.31=7.87/0.55=40K-Sat-H2-TCP-PLT/40K-MPQUIC_Psbol+WRR-PLT. This means that sending 8 x40KB objects over Sat using H2/TCP takes 14.31 times more than the same transfer using MPQUIC with PSBOL+WRR on the Sat + ADSL links.

So MPQUIC+PSBOL+WRR on a Sat + ADSL link is 14.31 times more efficient than H2/TCP when used on the satellite link alone.

Table 5 MPQUIC experimental results

SaT5G (761413) D4.3 May 2020

Uses cases for sequential mode in HTTP2 are chat, gaming for short objects and video streaming web download for long objects. Video streaming protocols make the client requesting successive chunks (of several hundred of KB each) and the server adapting the quality of the transmission of the next chunk to the current network capacities.

On the Satellite (50Mbps, 600ms) + ADSL (10Mbps, 20ms) tested network, the results in Table 6 show that in sequential mode, MPQUIC with PSBOL+WRR:

For short objects: has comparable performance as ADSL alone, i.e. is 4 to 20 times faster than Satellite alone and is 4 to 10times faster than MPQUIC with WRR,

For long objects: has comparable performance as MPQUIC WRR and improves the performance compared to Satellite or ADSL alone by providing bandwidth aggregation of the two links.

5.2.4.2 Parallel mode The experimental results for parallel traffic mode are shown in Table 7. In this table are reported the ratios between the related Page Load Time in the Table 5 and the MPQUIC PSBOL Page Load Time.

For example, 8.78 for Sat/H2/TCP means that a multi-link transfer using MPQUIC with PSBOL+WRR is 8.78 more efficient than H2/TCP on the Satellite link used alone when sending 8 40KB objects in parallel.

One can see the benefit of parallel traffic: HTTP 4K objects are sent in 8.78 seconds instead of 14.31.

Uses cases for parallel are mainly http2 web browsing. The Mix1 and Mix2 cases are representative of home pages of popular web sites and result from an analysis using Firefox debug tools that give objects sizes and scheduling timing information. On the Satellite (50Mbps, 600ms) + ADSL (10Mbps, 20ms) tested network, the results in Table 7 show that for parallel mode, MPQUIC with PSBOL+WRR:

For short objects: is significantly less performant than ADSL alone, is 3 to 9 times faster than Satellite alone and has comparable performance than MPQUIC with WRR. The less important performances gains compared to ADSL alone are due to the fact that the MPQUIC implementation does some packets retransmissions on the high delay satellite link if available. If such packets belong to short objects then the object transmission time will be increased at

Table 6 Sequential Mode experimental results

Table 7 Parallel Mode experimental results

SaT5G (761413) D4.3 May 2020

least to the delay of the satellite link causing a performance degradation. This behaviour is not observed when sending short objects in sequential mode because less or no retransmissions occur,

For long objects: has comparable performance as MPQUIC WRR and improves the performance compared to Satellite or ADSL alone by providing bandwidth aggregation of the two links,

For Mix1 Mix2 cases: is 2 or 3 times more performant than TCP or QUIC on the Satellite link used alone .This is due to bandwidth aggregation despite the retransmission of some packets of short objects on the high delay satellite link. For these mixed cases, MPQUIC with PSBOL + WRR has comparable performance than TCP or QUIC on ADSL alone. This is due to the limited bandwidth of the ADSL link and the moderate size of the exchanged objects. MPQUIC with PSBOL+ WRR has a little performance increase (15%) due to the limited object’s size the transfer times are short and the time where the aggregated is fully available is even shorter due to the slow start phase. Of course the penalty due to the retransmission of some packets belonging to short packets on the satellite link are also here and limit the PSBOL+ WRR performance increase.

SaT5G (761413) D4.3 May 2020

6 Standardization

The concepts, procedures and solution of traffic steering, switching and splitting evoked throughout the present document are mainly based on 3GPP specifications.

The study item AT3S and its related TR 23.793 [68] clearly identifies the following scope for the study of multi-link management:

- How the 5GC and the 5G UE can support multi-access traffic between 3GPP and non-3GPP accesses.

- How the 5G Core network and the 5G UE can support multi-access traffic switching between 3GPP and non-3GPP accesses.

- How the 5G Core network and the 5G UE can support multi-access traffic splitting between 3GPP and non-3GPP accesses (multi access PDU session).

- How the multi-access traffic steering, switching and splitting (AT3S) can be taken into account by the charging framework.

However, the study item does not cover the case of traffic steering, splitting and switching at the backhaul (only access is addressed).

Based on the work performed in SaT5G project there was a clear objective to propose the inclusion of the backhaul traffic management over multiple accesses to 3GPP work plan. In the frame of 3GPP SA2, a contribution regarding the topic has been prepared, submitted, and presented by the project partners in the frame of the satellite specific study item FS_5GSAT_ARCH in 3GPP SA2. The final contribution accepted (tdoc 3GPP SA2 S2-1902443: “Key issues Backhaul multilink”), thus introduces the study of multi-link backhaul as presented in Figure 6-1.

UE (R)AN 5GC Link 2

Link 1

Figure 6-1 Multi-link backhaul over two links as introduced in 3GPP

The following scope has been agreed for the study:

What are the impacts (if any) on the 5GS when introducing backhaul multi-connectivity with satellite link?

What is the finest backhaul traffic granularity which can be managed on N2, N3 interfaces (e.g. per-UE, per-QoS Flow)?

What are the criteria for steering, switching, and splitting the traffic towards a specific backhaul link and where the decision is taken?

Should the configuration of the protocols on N2, N3 be adapted in order to allow the support of satellite link (e.g. support of high latency)?

The analysis of these topics will continue in 3GPP and will be part of the technical report of the study, the TR 23.737.

SaT5G (761413) D4.3 May 2020

7 Conclusion

To sum up, on a 5G network satellite backhauling has several serious drawbacks for interactive exchanges.

Indeed, using an hybrid backhauling network with a low bandwidth and low latency terrestrial link combined with a high bandwidth and high latency satellite link improves significantly the performances provided if a multipath protocol such as MPQUIC with a good Path Selection Algorithm are used.

Current multipath protocols use Path selections algorithms such as Round Robin or Offload that provide bandwidth aggregation. That is fine for download exchanges but under-optimal for interactive operations because some packets are sent on the unwanted high latency link.

Path Selection Based On Object Length where short objects are sent on the low latency link and long objects on the two links using Round Robin or Offload solves the issue for interactive exchanges while leveraging the bandwidth aggregation capacity.

The experimentation based on MPQUIC with PSBOL+ WRR path selection outperforms the use of a Satellite or ADSL link by itself or the use of the two links with MPQUIC + RR for all types of applications, including HTTP2 Web browsing.

The WP5.3 implementation of those multi-link prototypes demonstrates that the concept of multi-link backhauling is possible with the multi-link protocol done in the client and the server and the multi-link interfaces placed on remote equipment of the backhauling network.

In this study we found or confirmed that:

Connecting UE by satellite alone on a 5G network has huge drawbacks for interactive exchanges due to the high satellite delays,

Using traffic splitting on multilink networks using satellite links along with low delay terrestrial links can solve the above issue provided that we use appropriate path selection algorithms taking in account the size of the exchanged objects in order to steer the packets belonging to short objects on low delay links and to split the packets belonging to long objects on all the links,

Using the concept of Hybrid backhauling is an elegant and useful way to integrate the satellite in 5G networks,

The BT3S concept instantiated in a Hybrid terminal and Gateway provides an MPTCP proxy supporting multilink to non AT3S UE and hosts connected to the UE or to the hybrid terminal and a policy based routing function to route the multipath subflows issued by the UE (AT3S or not) or by hosts routed by the UE or the hybrid terminal,

The Standard 5G multilink procedures AT3SS need to be extended in order to be able to leverage remote network interfaces such as the hybrid backhauling network ones, and to support efficient Path Selection Algorithms based on objects sizes.

There are 3 directions that should be further investigated:

AT3S extensions to support the use in splitting protocols of several subflows on the 5G interface and the use of Path Selection algorithms based on the size of the exchanged objects,

In each multilink protocol (MPTCP, MPQUIC), how to avoid retransmissions of packets belonging to short objects on high delay links,

Use the BT3S concept in order to fully integrate satellite links in 5G networks.

SaT5G (761413) D4.3 May 2020

Bibliography

[1] Broadband Forum, TR-348, Hybrid Access Broadband Network Architecture, 2016.

[2] Broadband Forum, WT-378, Nodal Requirements for Hybrid Access Broadband, 2019.

[3] IETF RFC 6824, TCP Extensions for Multipath Operation with Multiple Addresses, Network Working Group, January 2013.

[4] IETF RFC 3135, Performance Enhancing Proxies Intended to Mitigate Link-Related Degradations, Network Working Group, June 2001.

[5] IETF RFC 3449, TCP Performance Implications of Network Path Asymmetry, Network Working Group, December 2002.

[6] Cisco, https://www.cisco.com/c/en/us/products/collateral/routers/2800-series-integrated-services-routers-isr/solution_overview_c07-525404.html.

[7] ETSI TR 102 676, Satellite Earth Stations and Systems (SES); MBroadband Satellite Multimedia (BSM); Performance Enhancing Proxies (PEPs), version 1.1.1, November 2019.

[8] 3GPP TR 23.793, Study on access traffic steering, switch and splitting support in the 5G system architecture, release 16, version 16.0.0, Technical Specification Group Services and System Aspects, December 2018.

[9] J. Hwang and J. Yoo, Packet scheduling for Multipath TCP, Sapporo, Japan: 7th International Conference on Ubiquitous and Future Networks (ICUFN 2015), pp. 177–179, July 2015.

[10] K. W. Choi, Y. S. Cho, J. W. Lee, S. M. Cho, J. Ch, Optimal load balancing sheduler for MPTCP-based bandwidth aggregation in heterogeneous wireless environments, Computer Communications, vol. 112, pp. 116-130, November 2017.

[11] F. Rebecchi, M. Dias de Amorim, V. Conan, A. Passarella, R. Bruno and M. Conti, Data Offloading Techniques in Cellular Networks: A Survey, IEEE Communications Society, Institute of Electrical and Electronics Engineers, 17(2), pp. 580-603, 2015.

[12] BATS, FP7-ICT-2011-8, 2014.

[13] VITAL, H2020-ICT-2014-1, 2018.

[14] L. Y.-s, N. E. M., T. D. and G. R. J., ECF: An MPTCP Path Scheduler to Manage Heterogeneous Paths, Incheon, South Korea: 13th ACM International Conference on Emerging Networking EXperiments and Technologies (CoNEXT), pp. 147–159, December 2017.

[15] ETSI TR 103 124, Satellite Earth Stations and Systems (SES); Combined Satellite and Terrestrial Networks scenarios, version 1.1.1, July 2013.

[16] Sebastien Henri and Patrick Thiran, Optimal Number of Paths with Multipath Routing in Hybrid Networks, Chania, Greece: 19th IEEE International Symposium on A World of Wireless, Mobile and Multimedia Networks (WoWMoM’18), June 12-15, 2018.

[17] M. Tarique, K. E. Tepe, S. Adibi, and S. Erfani, Survey of Multipath Routing Protocols for Mobile Ad-hoc Networks. Journal of Network and Computer Applications, 2009.

[18] J. N. Al-Karaki and A. E. Kamal, Routing Techniques in Wireless Sensor Networks: a Survey, IEEE Wireless Communications, 2004.

[19] J. Tsai and T. Moors, A review of Multipath Routing Protocols: From Wireless Ad Hoc to Mesh Networks, ACoRN ECR Workshop on Multihop Wireless Networks, volume 30, 2006.

[20] IETF RFC 2992, Analysis of an Equal-Cost Multi-Path Algorithm, Network Working Group, November 2000.

SaT5G (761413) D4.3 May 2020

[21] P. Narvaez, K. Y. Siu, Efficient Algorithms for Multi-Path Link State Routing, ISCOM'99, Kaohsiung, Taiwan, 1999.

[22] H. S. Palakurthi, Study of Multipath Routing for QoS Provisioning, Oct. 2001.

[23] S. Vutukury and J.J. Garcia-Luna-Aceves, MDVA: A Distance-Vector Multipath Routing Protocol, In Proceedings of the INFOCOM, 2001.

[24] Srinivas Vutukury, Multipath routing mechanisms for traffic engineering and quality of service in the Internet, Santa Cruz: Ph.D thesis, University of California, March 2001.

[25] Hari Shri Palakurthi, Study of Multipath Routing for QoS Provisioning, Oct. 2001.

[26] IETF RFC 2991, Multipath Issues in Unicast and Multicast Next-Hop Selection, Network Working Group, November 2000.

[27] S. V. a. J. Garcia-Luna-Aceves, A Traffic Engineering Approach based on Minimum-Delay Routing, Las Vegas, Nevada, USA: Proceedings of IEEE ICCCN 2000, October 2000.

[28] E. Dinan, D.O.Awduche, and B.Jabbari, Analytical framework for dynamic traffic partitioning in MPLS networks, ICC’2000.

[29] Yongho Seok, et al, Dynamic constrained multipath routing for MPLS networks, Computer Communications and Networks 2001, pp. 348- 353, Oct. 2001.

[30] Yongseok Lee, et al, A constrained multipath traffic engineering scheme for MPLS networks, ICC 2002, April. 2002.

[31] IETF RFC 6182, Architectural guidelines for multipath TCP development, Network Working Group, Mar. 2011.

[32] IETF RFC 6356, Coupled congestion control for multipath transport protocols, Network Working Group, October 2011.

[33] O. Mehani, R. Holz, S. Ferlin and R. Boreli, An early look at Multipath TCP deployment in the wild, Paris, France: 6th ACM International Workshop on Hot Topics in Planet-Scale Measurement (HotPlanet), pp. 7–12., 2015.

[34] S. Barré, C. Paasch and O. Bonaventure, Multipath TCP: From Theory to Practice, Louvain-la-Neuve, Belgium: ICTEAM, Université catholique de Louvain, September 2014.

[35] N. Keukeleire, N. Hesmans and O. Bonaventure, Increasing broadband reach with Hybrid Access Networks, Louvain-la-Neuve, Belgium: Tessares, July 2019.

[36] B. Kimura, D. Lima and A. Loureiro, Alternative scheduling decisions for Multipath TCP, IEEE Communications Letters, vol. 21, no. 11, pp. 2412–2415, Nov. 2017.

[37] H. Shi; Y. Cui; X. Wang; Y. Hu; M. Dai, F. Wang and K. Zheng, STMS: Improving MPTCP throughput under Heterogeneous Networks, Boston, Massachusetts, USA: USENIX Annual Technical Conference (USENIX ATC), pp. 719-730, Jul. 2018.

[38] L. s., E. M. Nahum, D. Towsley and R. Gibbens, ECF: An MPTCP Path Scheduler to Manage Heterogeneous Paths, Incheon, South Korea: 13th ACM International Conference on Emerging Networking EXperiments and Technologies (CoNEXT), pp. 147–159, December 2017.

[39] N. Kuhn; E. Lochin; A. Mifdaoui; G. Sarwar and O. Mehani; and R. Boreli, DAPS: Intelligent delay-aware packet scheduling for multipath transport, IEEE International Conference on Communications (ICC). IEEE, pp. 1222–1227, 2014.

[40] ETSI TR 103 272, Satellite Earth Stations and Systems (SES); Hybrid FSS satellite/terrestrial network architecture for high speed broadband access, version 1.1.1, March 2015.

[41] ETSI TR 103 351, Satellite Earth Stations and Systems (SES); Multi-link routing scheme in hybrid access network with heterogeneous link, version 1.1.1, July 2017.

SaT5G (761413) D4.3 May 2020

[42] Olivier Bonaventure and SungHoon Seo., Multipath tcp deployments, IETF Journal, 12(2):24–27, URL: https://www.ietfjournal.org/multipath-tcp-deployments/., 2016.

[43] Quentin De Coninck and Olivier Bonaventure, Observing network handovers with multipath TCP. In Proceedings of the ACM SIGCOMM 2018 Conference on Posters and Demos, Budapest, Hungary: SIGCOMM 2018, URL: https://multipath-QUIC.org/multipathtester/2018/08/28/sigcomm-poster.html, August 20-25, 2018.

[44] IETF RFC 4960, Stream Control Transmission Protocol, Network Working Group, September 2007.

[45] M. Amend, MP-DCCP for enabling transfer of UDP or IP traffic over multiple data paths in multi-connectivity networks, IETF presentation, March 2019.

[46] QUIC for SATCOM, draft-kuhn-quic-4-sat-00 (IETF work in progress).

[47] E. Kohler, M. Handley, and S. Floyd, Designing DCCP: Congestion Control Without Reliability, ACM SIGCOMM, 2006.

[48] R. Stewart, RFC 4960: Stream Control Transmission Protocol (SCTP), Internet Engineering Task Force (IETF), 2007.

[49] M. Nowlan; N. Tiwari; J. Iyengar; S. Amin and and B. Ford, Fitting Square Pegs Through Round Pipes: Unordered Delivery Wire-Compatible with TCP and TLS, USENIX NSDI, 2012.

[50] L. Popa; A. Ghodsi; and I. Stoica, HTTP as the Narrow Waist of the Future Internet, ACM HotNets, 2010.

[51] J. Iyengar and M. Thomson, QUIC: A UDP-based multiplexed and secure transport, IETF, Working Draft: draft-ietf-QUIC-transport-08, https://tools.ietf.org/id/draft-ietf-QUIC-transport-08.txt, 12 2017.

[52] J. Iyengar and I. Swett, QUIC loss detection and congestion control, IETF, Working Draft: draft-ietf-QUIC-recovery-08, https://tools.ietf.org/id/draft-ietf-QUIC-recovery-08.txt, Dec. 2017.

[53] Y. Cui; T. Li; C. Liu; X. Wang; and M. Kühlewind, Innovating transport with QUIC: Design approaches and research challenges, IEEE Internet Computing, vol. 21, no. 2, pp. 72–76, Mar. 2017.

[54] A. Langley, A. Riddoch, A. Wilk, A. Vicente, C. Krasic, D. Zhang, F. Yang, F. Kouranov, I. Swett, J. Iyengar, J. Bailey, J. Dorfman, J. Roskind, J. Kulik, P. Westin, R. Tenneti, R. Shade, R. Hamilton, V. Vasiliev and W.-T. Chang, and Z. Shi, The QUIC transport protocol: Design and Internet-scale deployment, Los Angeles, California, USA: ACM Conference on Applications, Technologies, Architectures, and Protocols for Computer Communications (SIGCOMM), pp. 183–196, Aug. 2017.

[55] Jörg Deutschmann; Kai-Steffen Hielscher and Reinhard German, Satellite Internet Performance Measurements, IETF presentation , March 2019.

[56] N. Kuhn, E. Dubois, E. Lochin and L. Thomas, Google QUIC performance over a public SATCOM access, International Journal on Satellite Communications Networking, volume 37, number 6, pages 601-611, 2019.

[57] Cristian Mogildea, Jörg Deutschmann, Kai-Steffen Hielscher and Reinhard German, QUIC over satellite: introduction and performance measurements, 2019.

[58] G. Fairhurst; A. Custura; T. Jones, Measuring QUIC Dynamics over a High Delay Path, IETF presentation, https://datatracker.ietf.org/doc/slides-105-maprg-measuring-QUICdynamics-over-a-high-delay-path/, July 2019.

[59] Mamoutou Diarra; Luc Ottavj; Thierry Masson and Amine Ismail, 5G Hybrid Backhauling for Better QoE, Sorrento, Italy: Ka Conference, October 2019.

SaT5G (761413) D4.3 May 2020

[60] Q. D. Coninck and O. Bonaventure, Multipath Extension for QUIC, IETF, Working Draft: deconinck-multipath-QUIC-00, https://datatracker.ietf.org/doc/html/draft-deconinck-multipath-QUIC-00, Oct. 2017.

[61] T. Viernickel; A. Froemmgen; A. Rizk; B. Koldehofe and R. Steinmetz, Multipath QUIC: A deployable multipath transport protocol, IEEE International Conference on Communications (ICC), pp. 1–7, May 2018.

[62] Q. De Coninck and O. Bonaventure, Multipath QUIC: Design and evaluation, Incheon, South Korea: 13th ACM International Conference on Emerging Networking EXperiments and Technologies (CoNEXT), pp. 160–166., Dec. 2017.

[63] N. Kuhn; E. Stephan; J. Border; G. Fairhurst, QUIC Over In-sequence Paths with Different Characteristics, IETF presentation, https://datatracker.ietf.org/doc/slides-105-panrg-quicover-in-sequence-paths-with-different-characteristics, July 2019.

[64] 3GPP TS 23.502, Procedures for the 5G System (5GS); Stage 2, 3GPP, Technical Specification Group Services and System Aspects, release 16, version 16.2.0 , 09 2019.

[65] 3GPP TS 23.503, Policy and Charging Control Framework for the 5G System (5GS); Stage 2, 3GPP, Technical Specification Group Services and System Aspects, release 16, version 16.2.0, 09 2019.

[66] ETSI TR 103 611, Seamless integration of sat and/or HAPS systems into 5G system, 2018.

[67] 3GPP TS 23.501, System Architecture for the 5G System (5GS); Stage 2, 3GPP, Technical Specification Group Services and System Aspects, release 16, version 16.2.0 , 09 2019.

[68] 3GPP TR 23.793, Study on access traffic steering, switch and splitting support in the 5G system architecture, 3GPP, Technical Specification Group Services and System Aspects, release 16, version 16.0.0 , 12 2018.

[69] G. Lee and J. Choi, A Survey of Multipath Routing for Traffic Engineering, Information and Communications University, Korea, 2002.

[70] IETF RFC 5533, Shim6: Level 3 Multihoming Shim Protocol for IPv6, Network Working Group, June 2009.

[71] K. W. Choi, Y. S. Cho, J. W. Lee, S. M. Cho, J. Choi and e. al., Optimal load balancing scheduler for MPTCP-based bandwidth aggregation in heterogeneous wireless environments, Computer Communications, vol. 112, pp. 116–130, November 2017.