Presented by

Date

Event

SFO15-TR6-A: Openstack Testing on ARM64

Clark LaughlinClark Laughlin

Wednesday 23 September 2015

SFO15

Agenda1. Introduction2. Progress since HKG153. Current testing effort

a. Why aren’t we running an Openstack 3rd-party CI setup?

b. openstack.linaro.org & current CIc. results

4. Issues and Next Steps5. Questions?

Openstack

Hopefully know what Openstack is.

If not…http://www.openstack.org/software/

Devstack

A set of scripts for deploying an Openstack developer environment. Makes many assumptions, and deploys a working, non-production Openstack deployment on a single node.

http://docs.openstack.org/developer/devstack/

Tempest

Integration test suite for verifying an existing Openstack deployment● API tests, scenarios (no benchmarking)● Running as part of a Devstack deployment is much

easier -- Devstack generates the (mostly) proper Tempest configuration

http://docs.openstack.org/developer/tempest/

Since HKG15...

Since HKG15...March 2015 September 2015

Distro/Version Ubuntu/Trusty Trusty Wily

Tests Run 713 1521 1522

Pass 470 (66%) 1431 (94%) 1432 (94%)

Fail 51 (7%) 23 (1.5%) 23 (1.5%)

Skip 192 (27%) 5 (0.3%) 5 (0.3%)

Ignored No analysis done 62 63

Not tested cinder, neutron

Since HKG15...

enabled cinder, neutron

switched to kernel 4.1.x

enabled missing CONFIG_*

virtio-scsi, cirros

Current Testing - Why Not What We Originally Planned?

We did go through the process of creating an Openstack 3rd-Party CI system as we had planned and presented at HKG15.

So why aren’t we using it?

Current Testing - Why Not What We Originally Planned?

● Requires more ARM64 nodes than we had available to scale

● Not enough personnel to support… we wanted lunch breaks and sleep

● Openstack’s third-party CI processes are not geared towards platform testing, which is what we wanted to accomplish

So, what ARE we doing?

● Nightly Tempest runs on ARM64 systems○ Jenkins Matrix Job:

■ branch: master, stable/kilo (latest released stable)

■ os distro: Ubuntu■ os version: trusty, vivid, wily

○ Devstack & Tempest deployed and run by LAVA job

So, what ARE we doing?

Results reported publicly:

http://openstack.linaro.org

Openstack.linaro.org: Overall Summary

Openstack.linaro.org: Historical Graph

Openstack.linaro.org: Single Test

Failing Tests

● Depending on the test combination, ~19-28 tests currently failing

● Many failures we have eliminated have been due to Tempest configuration or the cloud image we were using

● Causes of remaining failures have not been root-caused yet

Skipped Tests

● Expected:○ Depending on the test combination, ~60-90 tests are

expected to be skipped for various reasons○ Various reasons:

■ subsystem needed by test not installed, i.e. ironic, nova-network

■ tests disabled due to known Openstack bugs■ feature disabled, i.e. VNC console

Skipped Tests

● Unexpected:○ Currently, only about 5 tests are being skipped○ 3 are due to Tempest configuration settings that still need

to be changed○ 2 (EC2 tests) are due to the cloud image we are using not

having ami/aki/ari manifest

Issues And Next StepsIssue Next Step(s)

Publicising Results - since we are not running as an official 3rd-party CI, our results are not readily visible to developers

Continue work with Openstack CI and Third-Party Testing Teams to determine the best way to share results

Hardware - currently only utilizing APM Mustang boards

Two HP Moonshot enclosures with 40 m400 cartridges are now available. Work to integrate these into the test environment.

Will you provide your hardware to include in our test environment?

Issues And Next Steps #2Issue Next Step(s)

Testing is not against a production deployment with HA/redundancy/failover -- everything is running on a single node

This is standard practice for normal Openstack CI. However, we are looking into testing against more production-quality deployments.

Only testing on Ubuntu Incorporate additional OS distros / versions● Debian● CentOS

Failure count still > 0 Continue to root-cause remaining failures

PresentationFinished.

Any Questions?

Presentation Finished.

Any questions?