bucketbenchBenchmarking Container Runtime Performance

Phil EstesSenior Technical Staff, IBM Cloud; Office of the CTOArchitecture Technical Team, Containers

@estesp

But...

Standard container lifecycle operations are not sufficient for our performance guarantees!

Cannot “docker build”, “docker run”, “docker rm” on each function invocation..

Containers make good sense as the function invocation

vehicle.

{DOCKER CE

CONTAINERD

RUNC

We Have Options!

Docker Engine architectureDocker 1.11 and above; April 2016-current

dockerComplete container engine with lifecycle

management, orchestration, remote API (daemon model), plugin support, SDN networking, image

building, image registry/local cache

management.

containerdHigh-performance,

standards-based lightweight container

runtime with gRPC API, daemon model. 1.0 contains complete lifecycle and image

management in Q42017.

runcOpen Container Initiative (OCI)

compliant implementation of the runtime specification. Lightweight container executor; no network,

image registry or image creation

capability.

https://github.com/estesp/bucketbench

A Go-based framework for benchmarking container lifecycle operations (using concurrency and load) against docker, containerd (0.2.x and 1.0), and runc.

The YAML file provided via the --benchmark flag will determine whichlifecycle container commands to run against which container runtimes, specifyingiterations and number of concurrent threads. Results will be displayed afterwards.

Usage: bucketbench run [flags]

Flags: -b, --benchmark string YAML file with benchmark definition -h, --help help for run -s, --skip-limit Skip 'limit' benchmark run -t, --trace Enable per-container tracing during benchmark runs

Global Flags: --log-level string set the logging level (info,warn,err,debug) (default "warn")

HOW C

AN WE

BENCHMARK

VARIOUS C

ONTAINER

RUNTIME O

PTIONS? examples/basic.yaml

name: BasicBenchimage: alpine:latestrootfs: /home/estesp/containers/alpinedetached: truedrivers: - type: Docker threads: 5 iterations: 15 - type: Runc threads: 5 iterations: 50commands: - run - stop - remove

Goals- Assess runtime stability under significant load/concurrency- Benchmark operational throughput of a container runtime

Table shows the rate of operation sequences per second. * indicates errors.---------------------------------------------------------------------------------------------------------------------------- Iter/Thd 1 thrd 2 thrds 3 thrds 4 thrds 5 thrds 6 thrds 7 thrds 8 thrds 9 thrds 10 thrds 11 thrds 12 thrds 13 thrds Limit 1000 651.3 829.4 834.5 809.6 827.6 848.6 774.8 843.2 800.3 839.2 804.2 806.7 813.0 DockerBasic 15 1.99 2.44 3.02* 3.24* 3.59* 3.90* 4.07*DockerPause 15 10.22 13.53 15.67 17.69 19.18 19.11 18.56 DockerFull 15 1.66 2.18* 2.69* 3.05* 3.21* 3.36* 3.63*ConBasic 50 2.90 4.95 6.54 7.49 8.10 8.33 8.65 9.02 9.25 9.17 9.43 9.22 9.25 RuncBasic 50 2.90 5.26 7.37 8.61 9.61 11.07 11.68 12.44 13.56 13.65 14.11 14.29 13.97

Caveats- Container configuration can greatly vary runtime performance- Direct comparison of runtimes not that valuable

Architecture

Two key interfaces: Driver

Drives the container runtime Bench

Defines the container operations and provides results/statistics

type Driver interface {

Type() Type

Info() (string, error)

Create(name, image string, detached bool, trace bool) (Container, error)

Clean() error

Run(ctr Container) (string, int, error)

Stop(ctr Container) (string, int, error)

Remove(ctr Container) (string, int, error)

Pause(ctr Container) (string, int, error)

Unpause(ctr Container) (string, int, error)

}

type Bench interface {

Init(driverType driver.Type, binaryPath,

imageInfo string, trace bool) error

Validate() error

Run(threads, iterations int) error

Stats() []RunStatistics

Elapsed() time.Duration

State() State

Type() Type

Info() string

}

Driver implementations support:

docker, containerd (1.0 via gRPC Go client API; 0.2.x via `ctr` binary), and runc today

Can easily be extended to support any runtime which can implement the Driver interface

Go tools: pprof, trace, block prof..

Also useful: strace, flame graphs..

API overhead, libnetwork setup/teardown, & metadata sync/update (locking) all add to differential from runc “bare” container start performance

Filesystem setup also measurable for large # of layers, depending on storage backend

Network namespace creation/deletion has significant impact under load● 300ms (and higher) delay in network spin lock under multi-threaded contention● Known issue:

http://stackoverflow.com/questions/28818452/how-to-identify-performance-bottleneck-in-linux-system-call-unshareclone-newnet

DISCOVERIES

Bucketbench: TODOs

1. Structured Output Format○ JSON and/or CSV output

2. Other Driver Implementations○ rkt? cri-o?○ Drive via CRI versus clients?

3. Integrate with Trace/Debug Tooling○ Randomized trace output (% of operations)○ “Real” performance metrics/tooling?

Thank You!1. Check out, critique, contribute to:

http://github.com/estesp/bucketbench

2. Connect with me to ask questions, or provide your own perspective and findings at @estesp on Twitter or

estesp@gmail.com