©2016 Couchbase Inc.©2016 Couchbase Inc.
Agenda
• Containers and their popularity • Orchestration Frameworks • Functional Testing at Couchbase with Docker• Interfacing with Docker • System testing with Sequoia Framework and Swarm •Demo
• Orchestrating Couchbase with Kubernetes • System testing with Swarm
• Best practices
4
©2016 Couchbase Inc.©2016 Couchbase Inc.
Containers and their popularity
•Customizable images • e.g. the same Couchbase Server build and OS, mix and match• Include all necessary libraries in the image
•One solution from Dev -> production• Deploy the same image in dev, preproduction and production across all platforms
• Packageable • can share a container for troubleshooting
• Lightweight • time to bring up a container with Couchbase• “clean slate”
• Popular• 500K+ Couchbase pulls from Dockerhub
5
©2016 Couchbase Inc.©2016 Couchbase Inc.
Orchestration and their frameworks
• Aggregation• Deploy apps implemented with multiple containers in pods
• Scaling• Increased the replication count across multiple servers• Computing resources per container
•Robustness• The ability to restart failing containers and hosts
• Services• Service discovery load balancing
6
©2016 Couchbase Inc.©2016 Couchbase Inc.
Functional Testing at Couchbase with Docker
• Couchbase Server Functional Testing Overview• Multiple test suites per component (e.g. Views, XDCR, N1QL etc.)• Each test suite:
• Contains 5 to 50 test cases• May use between 1 and 8 Couchbase Servers• Runs from 30 minutes to 10 hours
• Total of 150 test suites across 15 components• All regression suites runs in 12 hours
8
©2016 Couchbase Inc.©2016 Couchbase Inc.
Test Infrastructure - historical
Test client - takes as input a test suite
descriptor and cluster config
(Python)
Couchbase Server 1
Couchbase Server n
REST/CLI/memcached
VM
VM
Jenkins Job
©2016 Couchbase Inc.©2016 Couchbase Inc.
Challenges
• A multitude of Jenkins jobs (one per test suite)• Ordering is difficult• Test suite and cluster configuration implicitly stored in Jenkins jobs
• Dedicated VMs• Hard to maintain - if a VM is replaced with a new IP many Jenkins jobs must be edited• Fixed OS - testing with a different OS is cumbersome
©2016 Couchbase Inc.©2016 Couchbase Inc.
Docker solution
11
Test Suite Descriptors (Couchbase)
Test Suite Dispatcher
Test Suite Executor
Jenkins Job (multiple instances dependent on hardware capacity)
Couchbase Server 1 on Docker
Test Client
Test Suite N
Couchbase Server n on Docker
Couchbase Server 1 on Docker
Test Client
Test Suite 1
Couchbase Server n on Docker
©2016 Couchbase Inc.©2016 Couchbase Inc.
Couchbase Functional Test Suites and Kubernetes (Work in progress)
• Test suites are heterogeneous - differing number of Couchbase servers depending on the suite• Test suite runs are not persistent - when they are done the containers are not reused• Test client and Couchbase servers pod configuration:
• in 1 pod, or • spread across pods?• Pods per common cluster configs?
• Kubernetes Jobs may be a solution:• Job per test suites? Auto generated• Control parallelism?
©2016 Couchbase Inc.©2016 Couchbase Inc.
Requirements
15
Topology Changes
Data Loading
Query
• Building a framework for system testing
• JSON and binary documents with CRUD operations
• Map/Reduce, N1QL, Full Text Search, with index operations
• Add and remove servers, rebalance and failover cluster
©2016 Couchbase Inc.©2016 Couchbase Inc.
The Good News
16
Topology Changes
Data Loading
Query
• No problem...we already have tools to perform these operations
• pillowfight, cbdocloader, spring, gideon
• cbq, rest_client, vegetta
• Couchbase_cli, rest_client
©2016 Couchbase Inc.©2016 Couchbase Inc.
The Good News
17
Topology Changes
Data Loading
Query
• No problem...we already have tools to perform these operations
Package Together and we’re done!
©2016 Couchbase Inc.©2016 Couchbase Inc.
The Bad News
18
Topology Changes
Data Loading
Query
• Problem...tools are written if different languages
GOC
PythonJava
Bash
©2016 Couchbase Inc.©2016 Couchbase Inc.
The Docker News
19
Topology Changes
Data Loading
Query
• That’s fine...free containers for everyone!
GOC
PythonJava
Bash
©2016 Couchbase Inc.©2016 Couchbase Inc.
The Docker News
20
Topology Changes
Data Loading
Query
• From Implementing to Interfacing
GOC
PythonJava
Bash
INTERFACE
©2016 Couchbase Inc.©2016 Couchbase Inc.
The Docker News
21
• For Example
Topology Changes
Data Loading
Query
TEST
• run pillowfight -U 172.23.100.10
• run cbq -script=’create primary index on default’
• run couchbase-cli rebalance -c 172.23.100.10
©2016 Couchbase Inc.©2016 Couchbase Inc.
More Requirements
22
• What if I want to test a different cluster?
Topology Changes
Data Loading
Query
TEST
• run pillowfight -U 172.23.100.10
• run cbq -script=’create primary index on default’
• run couchbase-cli rebalance -c 172.23.100.10
©2016 Couchbase Inc.©2016 Couchbase Inc.
More Requirements
23
• Just change IP’s?
Topology Changes
Data Loading
Query
TEST
• run pillowfight -U 10.1.2.10
• run cbq -script=’create primary index on default’
• run couchbase-cli rebalance -c 10.1.2.11
©2016 Couchbase Inc.©2016 Couchbase Inc.
More Requirements
24
• Scope Abstraction
SCOPE
Topology Changes
Data Loading
Query
TEST
• run cbq -script=’create primary index on default’
• run couchbase-cli rebalance -c {{.Orchestrator}}
• run pillowfight -U {{.Orchestrator}}
Orchestrator: 10.1.2.10
©2016 Couchbase Inc.©2016 Couchbase Inc.
More Requirements
25
• What if I wanted to test against a cluster with 100 vms?
SCOPE
Topology Changes
Data Loading
Query
TEST
• run cbq -script=’create primary index on default’
• run couchbase-cli rebalance -c {{.Orchestrator}}
• run pillowfight -U {{.NthDataNode 70}}
…DataNode[0]: 10.1.2.10 QueryNode[0]: 10.1.2.11 DataNode[98]: 10.1.2.110
©2016 Couchbase Inc.©2016 Couchbase Inc.
Houston
26
• How about 100 containers?
SCOPE
Topology Changes
Data Loading
Query
TEST
• run cbq -script=’create primary index on default’
• run couchbase-cli rebalance -c {{.Orchestrator}}
• run pillowfight -U {{.NthDataNode 70}}
…DataNode[0]: ??? QueryNode[0]: ??? DataNode[98]: ???
©2016 Couchbase Inc.©2016 Couchbase Inc.
Solution
27
• Provider Abstraction
SCOPE
Topology Changes
Data Loading
Query
TEST
• run cbq -script=’create primary index on default’
• run couchbase-cli rebalance -c {{.Orchestrator}}
• run pillowfight -U {{.NthDataNode 70}}
PROVIDER
©2016 Couchbase Inc.©2016 Couchbase Inc.
TADA!
28
• Sequoia Framework
Topology Changes
Data Loading
Query
• Container Pluggability = More Tests + Less Coding• Scope Portability = Define Once + Run Anywhere• Resource Scalability = Your Laptop || The Lab
©2016 Couchbase Inc.©2016 Couchbase Inc.
Test expression
• Tests are a series of docker commands
30
• docker run cbq -script=’create primary index on default’
• docker run couchbase-cli rebalance -c 172.10.10.2
• docker run pillowfight -U 172.10.10.2
©2016 Couchbase Inc.©2016 Couchbase Inc.
Test expression
• With sequoia we express this using YAML files
31
©2016 Couchbase Inc.©2016 Couchbase Inc.
• New Swarm Provider • Containers are encapsulated in services (networking, multi-host)
Scaling With Swarm
35
DOCKER SWARM
containers services
©2016 Couchbase Inc.©2016 Couchbase Inc.
•Spread Strategy• Default behavior is to evenly distribute containers across couchbase servers
Scaling With Swarm
36
Manager 1
Manager 2
Manager 3
CB CLCB
Manager 4
CB CLCB CB CLCB CB CLCB
©2016 Couchbase Inc.©2016 Couchbase Inc.
•Spread Strategy• Default behavior is to evenly distribute containers across couchbase servers
Scaling With Swarm
37
Manager 1
Manager 2
Manager 3
CB CLCB
Manager 4
CB CLCB CB CLCB CB CLCB
©2016 Couchbase Inc.©2016 Couchbase Inc.
•Adding Constraints• Apply Labels for workload isolation
Scaling With Swarm
38
Manager 1
Manager 2
Manager 3
CB CB
Manager 4
CB CB CB CB CLCL
CL CL
CB CB CB
Server Server Server Client
• ‘docker node update --label-add zone=client manager-4’
©2016 Couchbase Inc.©2016 Couchbase Inc.
•Adding Constraints• Apply Labels for workload isolation
Scaling With Swarm
39
Manager 1
Manager 2
Manager 3
D D
Manager 4
D D D D CLCL
CL CL
I I Q
Server Server Server Client
©2016 Couchbase Inc.©2016 Couchbase Inc.
•Adding Constraints• Apply Labels for workload isolation
Scaling With Swarm
40
Manager 1
Manager 2
Manager 3
D D
Manager 4
D D D D CLCL
CL CL
I I Q
Server Server Server Client
©2016 Couchbase Inc.©2016 Couchbase Inc.
•Service Level Constraints• We can further isolate nodes by service
Scaling With Swarm
41
Manager 1
Manager 2
Manager 3
D D
Manager 4
I I CLCLQ
CL CL
D
Data Index Query Client
DDD
©2016 Couchbase Inc.©2016 Couchbase Inc.
Big Wins
•Multi-Host Load-Balanced Scalability with Isolation• Container automatically distributed evenly across Swarm nodes• Labeling allows service load balancing• Easy scale out• For FREE! We don’t have to rewrite a single test!
42
©2016 Couchbase Inc.©2016 Couchbase Inc.
Best Practices
• Isolate Isolate!• Consider CPU Pinning
• Set Quotas
• Share the memory
45
©2016 Couchbase Inc.©2016 Couchbase Inc.
Best Practices (continued)
• Be aware of when system calls reflect the host perspective e.g. available RAM, set time• Google and stackoverflow are your friends
46