1. 1. Presented by Date Event SFO15-TR6-A: Openstack Testing on ARM64 Clark LaughlinClark Laughlin Wednesday 23 September 2015 SFO15
2. 2. Agenda 1. Introduction 2. Progress since HKG15 3. Current testing effort a. Why arent we running an Openstack 3rd-party CI setup? b. openstack.linaro.org & current CI c. results 4. Issues and Next Steps 5. Questions?
3. 3. Openstack Hopefully know what Openstack is. If not http://www.openstack.org/software/
4. 4. Devstack A set of scripts for deploying an Openstack developer environment. Makes many assumptions, and deploys a working, non-production Openstack deployment on a single node. http://docs.openstack.org/developer/devstack/
5. 5. Tempest Integration test suite for verifying an existing Openstack deployment API tests, scenarios (no benchmarking) Running as part of a Devstack deployment is much easier -- Devstack generates the (mostly) proper Tempest configuration http://docs.openstack.org/developer/tempest/
6. 6. Since HKG15...
7. 7. Since HKG15... March 2015 September 2015 Distro/Version Ubuntu/Trusty Trusty Wily Tests Run 713 1521 1522 Pass 470 (66%) 1431 (94%) 1432 (94%) Fail 51 (7%) 23 (1.5%) 23 (1.5%) Skip 192 (27%) 5 (0.3%) 5 (0.3%) Ignored No analysis done 62 63 Not tested cinder, neutron
8. 8. Since HKG15... enabled cinder, neutron switched to kernel 4.1.x enabled missing CONFIG_* virtio-scsi, cirros
9. 9. Current Testing - Why Not What We Originally Planned? We did go through the process of creating an Openstack 3rd-Party CI system as we had planned and presented at HKG15. So why arent we using it?
10. 10. Current Testing - Why Not What We Originally Planned? Requires more ARM64 nodes than we had available to scale Not enough personnel to support we wanted lunch breaks and sleep Openstacks third-party CI processes are not geared towards platform testing, which is what we wanted to accomplish
11. 11. So, what ARE we doing? Nightly Tempest runs on ARM64 systems Jenkins Matrix Job: branch: master, stable/kilo (latest released stable) os distro: Ubuntu os version: trusty, vivid, wily Devstack & Tempest deployed and run by LAVA job
12. 12. So, what ARE we doing? Results reported publicly: http://openstack.linaro.org
13. 13. Openstack.linaro.org: Overall Summary
14. 14. Openstack.linaro.org: Historical Graph
15. 15. Openstack.linaro.org: Single Test
16. 16. Failing Tests Depending on the test combination, ~19-28 tests currently failing Many failures we have eliminated have been due to Tempest configuration or the cloud image we were using Causes of remaining failures have not been root-caused yet
17. 17. Skipped Tests Expected: Depending on the test combination, ~60-90 tests are expected to be skipped for various reasons Various reasons: subsystem needed by test not installed, i.e. ironic, nova-network tests disabled due to known Openstack bugs feature disabled, i.e. VNC console
18. 18. Skipped Tests Unexpected: Currently, only about 5 tests are being skipped 3 are due to Tempest configuration settings that still need to be changed 2 (EC2 tests) are due to the cloud image we are using not having ami/aki/ari manifest
19. 19. Issues And Next Steps Issue Next Step(s) Publicising Results - since we are not running as an official 3rd-party CI, our results are not readily visible to developers Continue work with Openstack CI and Third-Party Testing Teams to determine the best way to share results Hardware - currently only utilizing APM Mustang boards Two HP Moonshot enclosures with 40 m400 cartridges are now available. Work to integrate these into the test environment. Will you provide your hardware to include in our test environment?
20. 20. Issues And Next Steps #2 Issue Next Step(s) Testing is not against a production deployment with HA/redundancy/failover -- everything is running on a single node This is standard practice for normal Openstack CI. However, we are looking into testing against more production-quality deployments. Only testing on Ubuntu Incorporate additional OS distros / versions Debian CentOS Failure count still > 0 Continue to root-cause remaining failures
21. 21. Presentation Finished. Any Questions? Presentation Finished. Any questions?