HTTP/WebDAV synchronization protocol optimizations.
Piotr Mrowczynski
HTTP/WebDAV synchronization protocol optimizations.
- HTTP2 (https://github.com/owncloud/client/compare/http2)
- Bundling (https://github.com/owncloud/client/pull/5319)
- Request Scheduling (https://github.com/owncloud/client/pull/5440)
- Dynamic Chunking for new chunking algorithm (https://github.com/owncloud/client/pull/5368)
- Request Scheduling (https://github.com/owncloud/client/pull/5440)
- Prioritize by modification time (https://github.com/owncloud/client/pull/5349)
Scope of this talk
Current ownCloud WebDAV / HTTP1.1 implementation
- HTTP/1.1 without pipelining – head of line blocking
- HTTP/1.1 without pipelining – head of line blocking
- max 3-6 parallel connections (as in web)
- each file is single PUT / GET / DELETE / MKDIR / MOVE request within a single persistent (Keep-Alive Header) connection
Tim
e
MKCOL
...
Con. 1 Con. 2 Con. 6PUT PUT
Server can handle 100 requests in parallel at specific moment → client will use anyways max 6
- server can handle 100 requests in parallel at specific moment → client will bind to max 6
- server with concurrent syncs is overloaded with x6 opened connections (usually SSL)
- server can handle 100 requests in parallel at specific moment → client will bind to max 6
- server with many concurrent syncs is overloaded with opened connections
- Latency: Each file in separate connection has to waste time on latency (usually 15-300 ms)
Tim
e
Latency and transf
er
Latency
Serv
er-s
ide
Ope
ratio
ns
1000 files / 6 parallel = 167 lines of blocking → x 53 ms = 9s
x 320 ms = 53s
HTTP1, HTTP2 and BUNDLING
Tim
e
MKCOL
...
Con. 1 Con. 2 Con. 6PUT PUT
HTTP/1.1
Tim
e
MKCOL
...
Con. 1 Con. 2 Con. 6PUT PUT
HTTP/1.1
...
Con. 2BUN
DLE
Tim
e
MKCOL
Con. 1 Con. 6BUN
DLE
Con. 1
BUNDLING
Reduced latency gain
Tim
e
MKCOL
...
Con. 1 Con. 2 Con. 6PUT PUT
HTTP/1.1
Tim
e
PUTMax Parallel {
Con. 1
Bandwidth/Time Max Parallel Gap
Optimization target – request scheduling (will talk later)
HTTP 2 with ownCloud's
requests limitation
...
Con. 2BUN
DLE
Tim
e
MKCOL
Con. 1 Con. 6BUN
DLE
Con. 1
BUNDLING
Reduced latency gain
HTTP2 with ownCloud request limitation
If optimized by pumping more requests or binary data transfers: - might hide request-response latency as in bundling
- might utilize bandwidth
- only limited by server/database capability of accepting parallel files
HTTP2 - possible benefits for ownCloud
- The parallel multiplexed requests and response do not block each other.
HTTP2 - possible benefits for ownCloud
- The parallel multiplexed requests and response do not block each other.
- Optimized and faster encryption
HTTP2 - possible benefits for ownCloud
- The parallel multiplexed requests and response do not block each other.
- Optimized and faster encryption
- Binary framing – less errors, overhead and more
- Header compression
HTTP2 - possible benefits for ownCloud
- The parallel multiplexed requests and response do not block each other.
- Optimized and faster encryption
- Binary framing – less errors, overhead and more
- Header compression
- Flow control (separate from TCP flow control)
BUNDLING - possible benefits for ownCloud
- Files packed in group of requests, send over the network and single response is returned
- Above results in latency reduction and possible better network utilization
BUNDLING - possible benefits for ownCloud
- Files packed in group of requests, send over the network and single response is returned
- Above results in latency reduction and possible better network utilization
- Reduces PHP overhead (script is fired up once for whole the group instead of per file)
- PHP overhead – can be also reduced by optimizing server side for single requests
HTTP1 vs HTTP2 tests
HTTP1 vs HTTP2 testsFiles in parallel limitation
1000 files 1kB – total 1MB of data
CERNBOXGeneva, Switzerland Berlin, Germany
Melbourne, Australia
Measurement repeated 10 times using Smashbox Benchmarking Tool
SSD, 8GB RAM, 4x2,4GHz,
Openstack, 12GB RAM, 4x2.5Ghz
WiFi, 53 ms latency, Upl
76Mbit/s, Dow 79 Mbit/s
Ethernet, 320 ms latency, Upl 220 Mbit/s, Dow 1448 Mbit/s
HTTP1 vs HTTP2 testsFiles in parallel limitation
1000 files 1kB – total 1MB of dataMeasurement repeated 10 times using Smashbox Benchmarking Tool
Synchronization to CERNBox (EOS), Geneva, Switzerland
Protocol Parallel Limit Location Upload Time [s] Download Time [s]
HTTP1 6 53 ms, DE 115.9 +/- 39.6 58.7 +/- 17.9
HTTP2 98.8 +/- 34.8 53.8 +/- 19.2
HTTP1 6 320 ms, AU 230.1 +/- 27.2 151.4 +/- 44.1
HTTP2 186.7 +/- 27.6 213.0 +/- 24.1 (?)
HTTP1 100 320 ms, AU 209.8 +/- 21.0 129.4 +/- 21.7
HTTP2 39.6 +/- 11.7 42.6 +/ 9.2 (?)
Latency and WAN impact
-17s
-43s
HTTP1 vs HTTP2 testsFiles in parallel limitation
1000 files 1kB – total 1MB of dataMeasurement repeated 10 times using Smashbox Benchmarking Tool
Synchronization to CERNBox (EOS), Geneva, Switzerland
Protocol Parallel Limit Location Upload Time [s] Download Time [s]
HTTP1 6 53 ms, DE 115.9 +/- 39.6 58.7 +/- 17.9
HTTP2 98.8 +/- 34.8 53.8 +/- 19.2
HTTP1 6 320 ms, AU 230.1 +/- 27.2 151.4 +/- 44.1
HTTP2 186.7 +/- 27.6 213.0 +/- 24.1 (?)
HTTP1 100 320 ms, AU 209.8 +/- 21.0 129.4 +/- 21.7
HTTP2 39.6 +/- 11.7 42.6 +/ 9.2 (?)
1. Head of line blocking2. Server bookkeeping blocks upload
HTTP2 PipeliningClient still allows only max 6 connections
-170s
HTTP1 vs HTTP2 testsFiles in parallel limitation
1000 files 1kB – total 1MB of dataMeasurement repeated 10 times using Smashbox Benchmarking Tool
Synchronization to CERNBox (EOS), Geneva, Switzerland
Protocol Parallel Limit Location Upload Time [s] Download Time [s]
HTTP1 6 53 ms, DE 115.9 +/- 39.6 58.7 +/- 17.9
HTTP2 98.8 +/- 34.8 53.8 +/- 19.2
HTTP1 6 320 ms, AU 230.1 +/- 27.2 151.4 +/- 44.1
HTTP2 186.7 +/- 27.6 213.0 +/- 24.1 (?)
HTTP1 100 320 ms, AU 209.8 +/- 21.0 129.4 +/- 21.7
HTTP2 39.6 +/- 11.7 42.6 +/ 9.2 (?)
On latency 50ms from Berlin to Geneva, we got even 20s →50 Hz (files per second)
HTTP1 vs HTTP2 testsSSL Overhead
12 files 1kB – total 12kB of dataMeasurement repeated 10 times using Smashbox Benchmarking Tool
Synchronization to CERNBox (EOS), Geneva, Switzerland
Protocol Files Synced Location Upload Time [s] Download Time [s]
HTTP112 320 ms, AU
6.7 +/- 1.4 5.0 +/- 1.3
HTTP2 5.9 +/- 2.2 4.4 +/- 0.4
3-way-handshake and SSL optimization gain, connection reuse limit (only ?)
-1.8s
HTTP1 vs BUNDLING tests
HTTP1 vs BUNDLING tests1000 files 1kB – total 1MB of data
DAMKEN CLOUDNuremberg, Germany Berlin, Germany
Melbourne, Australia
Measurement repeated 10 times using Smashbox Benchmarking Tool
SSD, 8GB RAM, 4x2,4GHz,
Openstack, 12GB RAM, 4x2.5Ghz
WiFi, 37 ms latency, Upl
76Mbit/s, Dow 79 Mbit/s
Ethernet, 279 ms latency, Upl 220 Mbit/s, Dow 1448 Mbit/s
HTTP1 vs BUNDLING tests Files in parallel limitation
1000 files 1kB – total 1MB of dataMeasurement repeated 3 times using Smashbox Benchmarking Tool
Synchronization to Damken Cloud, Nuremberg, Germany
Protocol Bundled Files Location Upload Time [s] Download Time [s]
HTTP1 -37 ms, DE
155.1 +/- 0.4 41.0 +/- 0.8
Bundling 100 149.1 +/- 0.8 38.0 +/- 3.5
HTTP1 -
279 ms, AU
187.8 +/- 5.6 67.6 +/- 0.2
Bundling 100 158.4 +/- 3.2 66.0 +/- 1.3
Bundling 10 152.6 +/- 1.2 67.9 +/- 0.1
Latency Influence on HTTP1Bundling in this prototype works only for upload
-6s
-28s
HTTP1 vs BUNDLING tests Files in parallel limitation
100 files 1kB – total 100kB of data
1000 files to bundle →10 requests needed
100 files to bundle→6 requests needed (as number of connections)
It occurred that for 100 files sync time reduced: 20s → 16s in upload for 37ms latency
Requests Scheduling
“Wide and Narrow Pipe” problem with max 3-6 connections
2 MB/savailable
10 kB 10 kB 10 kB 10 kB10 kB
10 kB 10 kB
“Wide and Narrow Pipe” problem with max 3-6 connections
2 MB/savailable
5 MB 5 MB 5 MB
Better solution utilizing 6 connections
2 MB/savailable
5 MB 5 MB
10 kB 10 kB
10 kB 10 kB
Even better solution using HTTP/2 for fast and idle servers
Boosted using http2 prioritization? (E. Bocchi, Politecnico di Torino)
2 MB/savailable
5 MB 5 MB
10 kB 10 kB
10 kB 10 kB
10 kB 10 kB
10 kB 10 kB
10 kB 10 kB
10 kB 10 kB
10 kB 10 kB
10 kB 10 kB
10 kB 10 kB
10 kB 10 kB
10 kB 10 kB
10 kB 10 kB
TU Berlin → TU Berlin Test
20 x 5MB50 x 100kB
Folder A Folder B
Measurement repeated 10 times using Smashbox Benchmarking Tool
TU Berlin → TU Berlin TestOld implementation
Folder-wise, first folder with small, then folder with big
TU Berlin → TU Berlin TestNew implementation
Cross-folder, big files and small files at the same time
TU Berlin → TU Berlin Test
20 x 5MB 50 x 100kB
Upload: ~23s → ~20s Download: ~16s → ~13s
Measurement repeated 10 times using Smashbox Benchmarking Tool
- Small files don't block bigger ones
- Big files don't block smaller ones
Future using HTTP2, Dynamic Chunking and Scheduling?
Sending you 3 files
Ok, but I can handle 50 files at this moment
My dynamic chunk is now 20 MB.
Sending you 30 fileswhich fit into it
I have more users now
Reduce to 15 files
- Using HTTP2 PUSH in Discovery phase? (E. Bocchi, Politecnico di Torino)
- Max Parallel Negotiation
Take Away Message- Changing the way the requests are being send may significantly reduce the sync time (HTTP2, Bundling, Scheduling)
- If you want to test HTTP2 or any of the features, please contact me [email protected]
HTTP2 Bundling
Number of Connections 1 Max 6
Optimized transfer Optimized encryption, Binary stream, header compression, 1 connectionPHP overhead reduced, files
“buffered” in bundles on server
Number of files processed in parallel
Limited only by server capability and client OS/hardware – probably max
100Max 6
Observed improvement
CERNBox + EOS (1000 files)115s → 20s on 50ms lat.210s → 40s on 320ms lat.
6.7s → 5.9s for 10 files on 320ms lat
DamkenCloud (1000 files)155s → 149s on 37ms lat188s → 158s on 279ms lat
20s → 16s for 100 files on 37ms lat
Slide 1Slide 2Slide 3Slide 4Slide 5Slide 6Slide 7Slide 8Slide 9Slide 10Slide 11Slide 12Slide 13Slide 14Slide 15Slide 16Slide 17Slide 18Slide 19Slide 20Slide 21Slide 22Slide 23Slide 24Slide 25Slide 26Slide 27Slide 28Slide 29Slide 30Slide 31Slide 32Slide 33Slide 34Slide 35Slide 36Slide 37Slide 38Slide 39Slide 40