LAVA Multi-node Testing

Antonio Terceiro, Senthil Kumaran, Neil Williams, James Tunnicliffe & Fu Wei

LCE13 - Dublin, July 2013

Support for testing use cases across multiple devices

DISCLAIMER: none of the stuff you are about to see is in production yet.

→�You get an Unicode cat, though

● Scope and use cases● Proposed API● Demo (???)● Status and Future Plans● Feedback - open discussion

Session Outline

Scope and use cases

● test jobs involving multiple devices● devices may be of same type, or not - it

doesn't matter● devices get roles assigned to them● each device can be flashed with a different

test image

Scope (1)

● test writers can write any test they want.● LAVA provides information/synchronization

primitives● each device runs tests as it would normally

run, plus they can synchronize with each other using the API

● in the test results are aggregated into a single results bundle.

Scope (2)

● Requests 2 devices● Role 1: server● Role 2: client● Server:

○ install/start server application● Client:

○ install client application○ hits server, does benchmarks

Example 1: client/server interaction

● Requests 6 devices. Roles:○ database-server x 1○ web-backend x 2○ load-balancer x 1○ client x 2

● wiring up the application together is test writer's job

● provisioning/orchestration tools can be used since tests are shell scripts○ salt○ puppet○ chef○ etc

Example 2: multi-tier application

● the test suite for toolchain component X can be cleanly split in say 3 components, a, b, c

● Request 3 devices with roles, a, b, c● Node with role ${label} will download pre-

built binaries and run the test suite component labelled with ${label}

Example 3: parallel build/test run

● Physical setup:○ Phone connected through USB to x86 host○ Host configured to pass through the USB device to a

KVM device● Request one phone device and one KVM

device● Deploy latest Android build to phone● Boot Ubuntu build to KVM● Test that you can do ADB over USB

operations from Ubuntu to the phone

Example 4: ADB connectivity

● The LAVA team will provide the infrastructure and support

● Other teams can come up with whatever tests they need○ that are physically possible ;-)

● use cases that require specialized hardware setups require lab work and will be handled in a case-by-case basis.

Example n ...

Proposed API

device_group: [

{

"device_type": "highbank",

"role": "server",

"count": 1,

},

{

"device_type": "panda",

"role": "server",

"count": 1,

},

]

JOB submission (1)

device_group: [

{

"device_type": "galaxy-nexus",

"role": "phone",

"count": 1, "tags": "usb-pair-01"

},

{

"device_type": "kvm",

"role": "x86-box",

"count": 1, "tags": "usb-pair-01"

},

]

JOB submission (1.1)

Requesting a specific hardware setup

actions: [{

"command": "deploy_linaro_image","parameters": {

"image": "url://to/images/server.img"},"role": "server"

},[...]

]

JOB submission (2)

only applies to the devices

with the "server" role

actions: [// ...{

"command": "lava_test_shell","parameters": {

"testdef_urls": ["url://to/testdef.yaml"

]}

},// ...

]

JOB submission (3)

THIS DID

NOT CHA

NGE

# testdef.yamlmetadata: name: lava-test-shell-simple description: "description of this testdef" version: 1.0 format: "Lava-Test-Shell Test Definition 1.0"

run: steps: - lava-test-case test1 --result pass - lava-test-case test2 --result fail - lava-test-case test3 --shell ./test-script - [...]

LAVA-Test-Shell Review (1)

# testdef.yamlmetadata: name: lava-test-shell-simple description: "description of this testdef" version: 1.0 format: "Lava-Test-Shell Test Definition 1.0"

run: steps: - ./run-my-test-suite

LAVA-Test-Shell Review (2)

Set of programs in $PATH that are available for your LAVA-Test-Shell tests to call

lava-test-caselava-test-case-attachlava-test-run-attachlava-os-buildlava-installed-packageslava-install-packages

The LAVA Test Shell API

An extension of the existing LAVA Test Shell API that adds:

● Commands that provide contextual information

● Commands for communication and synchronization

The LAVA multi-node API

lava-role

Prints the role of the current device

# testdef.yamlrun:

steps:- ./run-$(lava-role).sh

lava-group

Prints the hostnames and roles in the group

$ lava-grouppanda01 clienthighbank01 backendhighbank02 backendhighbank03 load-balancer

lava-sync

Global synchronization across the group

# testdef.yamlrun:

steps:- ./setup-$(lava-role).sh- lava-sync setup-done- ./run-$(lava-role).sh

lava-send <msg-id> [k1=v1 k2=v2 ...]

Sends a message to the group

lava-wait <msg-id>

Waits for <msg-id> from some other node

lava-wait-all <msg-id>

Waits for <msg-id> from all other nodes

Other building blocks

Demo

Status and Future Plans

● Implementation in advanced stage● Started initial testing● Should be available in production at 2013.

XX

Status

● Exclusive lock on groups of devices○ "this 10 devices share a network bus. Even if I am

only using 5, I don't want the other 5 to be used simultaneously because that may affect the results of my networking benchmarks

● [...]

Future work

Feedback(Open discussion)

● Tell us about your testing use cases● Does lava-send <msg-id> k1=v1 k2=v2

look useful?● "What if I need a specific hardware

configuration for my tests?"● ...

Seed questions