LCU14 BURLINGAME

Milosz Wasilewski, LCU14

LCU14-509: Overview of QA reports

● Why are we doing it?● single place to check build and test status● manual and automated tests in one place● simple weekly/monthly reporting● bugs/known issues together with test results

QA Reports - overview

If the following conditions are met, the project is ‘green’:● build(s) successful● tests submitted to LAVA● all submitted tests produce results● all tests pass

Single build&test status

Single build&test status cont.

● Manual testing covers some areas that are hard to automate (UI, USB hotplug etc.)

● Till now:● Automated results were detached from automated● Reporting to MLs contained only manual test results● Changing test plan was painful

● From now:● Manual tests tied to build with clickable link● Manual and automated tests in one view● Test plan can be updated whenever there is a need (i.e. “won’t fix” bug appears)

Manual and automated test in one place

Manual and automated test in one place cont.

manual test results

automated test results

● So far reporting was manual● Reporting covered only manual test results● With qa-reports:

● reporting covers automated and manual tests● defects can be added to failed tests● reports can span across builds/devices● report is just an aggregated view of all results● additional comments can be added to the report (i.e. explaining the purpose of the

report or builds used)

Simple reporting

Simple reporting cont.

report description

automated test results

Simple reporting cont.

known issues extracted from all results

manual test results

● It’s possible to compare results between boards/builds

Result comparison

● Amount of data is huge● There is a lot of data duplication

● automated test results copied from LAVA● test definition versions copied from git repositories

● UI design isn’t perfect● Missing handy features:

● email notifications● more complex comparison● tracking of test case changes● measurement display● measurement graphs● ...

Issues

More about Linaro Connect: connect.linaro.org Linaro members: www.linaro.org/membersMore about Linaro: www.linaro.org/about/