Page 1
Software Test Automation
Texas InstrumentsSoftware Development Organization
September 2009
Page 2
Agenda
1. Goals of Common SDO test automation effort.
2. System Architecture / Layers
3. Test Setup
4. Test Plans
5. Roadmap
Page 3
1) Goals
Page 4
Goal of SDO Automation Effort
To establish a common integration and test infrastructure for development and test teams to use across SDO that leverages open source components
To enable customers, partners, and community to run regression tests and contribute tests
Page 5
Goals and Solutions
• Goal: Share same test suites/test cases across sites– Solution:
• Use open source test management system (Test Link) to store test suites.• Use single test database at all sites• Automate the test execution of most test suites/test cases.
• Goal: Open architecture to support multiple execution engines– Solution:
• Design a system that allows using different types of execution engines to suit user needs. • The execution engine type is part of the test case definition.
• Goal: Share hardware resources– Solution:
• Implement a common Resource Manager that allocates resources as needed
• Goal: Enable more people to run tests – Solution:
• Implement a web-based test management system that allows people to trigger common tests and get results at local or remote locations.
• Goal: Facilitate Continuous Integration & Test– Solution:
• Incorporate Build Execution Engines (BEE) into the test automation architecture so a Continuous-Integration system can be implemented.
• Code can be built and tested by developers and testers as often as desired and at any point in the development cycle.
• Goal: Maximize Open Source Software usage– Solution:
• Leverage existing open source tools as much as possible• The new test architecture uses TestLink and Staf.
Page 6
2) System Architecture
Page 7
High-Level Test Architecture
Test Master Controller
Build ExecutionEngine
Test Management System
Test Artifacts (Mysql)
ServiceProvider
Test ExecutionEngine
Test ExecutionEngine
ServiceProvider
Test ExecutionEngine
Site A
BuildExecutionEngine
Test Management System
Test Artifacts (Mysql)
Service Provider
Test ExecutionEngine
Test ExecutionEngine
ServiceProvider
Test ExecutionEngine
Site B
IP Network
IP Network
• Test Artifacts Databases must be synchronized on a periodic basis• The 3rd layer is the service-providers layer. TEE and BEE are just special Service Providers.• It should be possible to reuse Service Providers across sites if/when desired• Typically 1 TMS : 1 TMC 1 TMC : Many Service Providers
Test Master Controller
• Create and Manage Test Cases & Organize them into Test Plans.• Request Test Job
Execution.
• Manage Resources.• Dispatch & Monitor
Jobs.
• Execute Build & Test Jobs.
Page 8
Test Architecture: 3 Functional Layers
1) Test Management System. 2) Test Master Controller. 3) Service Providers
Test Projects
Test Management System
TI CustomFields*
Test Master Controller
TestLink (GPL) + Stafhttp://www.teamst.org/
Staf (EPL)http://staf.sourceforge.net/
Test Reports
Test Requirements
Test Execution Engine
…
Equipment Drivers
* Custom fields= Test Execution Engine, Test Execution Logic, Test Parameters,HW Assets, SW Assets
Staf + Vatf|Stax|iCOTS|TestRunner|Other
Test Cases
We
b F
ron
t-E
nd
Execution Logic
Resource Management Service
n
1
1
Data in XML
Data adapted to TEE
Build Execution Engine
…
Execution Logic
Data adapted to BEE
n
Dispatcher Service
Monitor Service
Test Plans
Page 9
Test Management Layer: Test Link
Test Link is an open source test management tool– It uses a MySQL database to hold test cases / test suites / test plans
– It runs on a server, and requires no proprietary tools to run
– It uses a browser as the user interface
– It does not natively have good support for automated test cases, but we are enhancing its automation support by integrating it with STAF
Page 10
Test Master Controller Layer: Staf
Staf is an open source, multi-platform, multi-language framework designed around the idea of reusable components.
– It allows Staf processes on different machines to communicate & initiate processes on each other
We will use it for:– Resource management
– Dispatching test requests for execution
– Tracking test execution
– Communication between layers of the system
Page 11
Service Providers: TEE / BEE Layer
The service providers layer creates builds and runs tests
Initial implementation:– Test Engine: VATF (Home-grown Test Execution Engine)
– Build Engine: File BEE (provides pre-built image)
Arago BEE (builds image using Arago/OE infrastructure)
Staf client will run on test execution systems to interface the TEE and BEE components to the TMC layer, and to perform resource registration / status
Page 12
3) Test Setups
Page 13
High-Level Test Setup
Page 14
TEE Test Setup details (USB)
Page 15
TEE Test Setup Details (OVQ)
Page 16
4) Test Plans
Page 17
Test Plans
• TI will offer Test Plans for individual components (i.e. PSP) and for integrated systems/solutions (i.e. DVSDK).
• TI will provide Test plans for different test scopes:– L1: Level 1 Sanity (a.k.a Smoke) tests focus on verifying that a feature or
component is operational. Each L1 test should take at most few minutes to complete and the whole L1 test plan should not take more than 12 hours to run. L1 test plan should be run as frequently as possible.
– L2: Level 2 tests focus on complete functional validation. The L2 Test plan may take few days to run and may involved manual testing.
– L3: Level 3 tests focus on measuring performance, stressing the system and exercising system use case scenarios. The L3 test plan may take few weeks to run and most likely required some manual testing.
Page 18
Test Plans cont.
• TI has test plans for many areas as shown in the next two slides, however they are not yet in TestLink format. TI is currently in the process of migrating existing test plans to TestLink.
• After migrating the test plans to TestLink, we will benefit from some of the features of the new system such as reuse same test logic at different locations, hardware resource management and continuous testing of software built using Arago.
• Existing automated test plans focus primarily on L1 and L2 testing. TI is working on adding more L3 tests as well as incorporating new open source test tools into the automated test plans.
Page 19
PSP Test Areas
• ATA
• Audio ALSA, OSS
• CIR
• EDMA
• Ethernet
• GPIO
• I2C
• Keyscan
• MMC
• SD, SDHC, SDIO
• NAND
• NOR
• RTC
• SPI
• Timer
• UART
• Uboot
• USB Host MSC, ISO, HID
• USB Slave MSC, CDC/RNDIS
• Video Fbdev Display
• Video V4L2 Display
• Video V4L2 Capture
• Video H3A
• Video Previewer/Resizer
• Video Face Detection
• WDT
Page 20
DVSDK Test Areas
• Applications:– Demos
– DVTB
– Gstreamer
• Codecs:– H.264 encoder
– H.264 decoder
– MPEG4 encoder
– MPEG4 decoder
– JPEG encoder
– JPEG decoder
– AAC encoder
– AAC decoder
– G711 encoder
– G711 decoder
– G729 encoder
– G729 decoder
– MP3 decoder
Page 21
Test Equipment & Drivers
• Drivers for the following Hardware has already been written and validated: – Generic Linux host driver (DM355, DM357, DM365, DM644x, DM6467, etc.) – APC Power Controller – Extron Video/Audio Switches – Extron USB Switches– Tascam DVD players – Pionner DVD players – Q-Master's Video Quality System – Video Clarity's Video Quality System – Generic PC Sound Card– Spectra Lab's Audio Quality System – Opera’s PESQ Speech Quality System.
• There are also some second-tier drivers that reuse HW drivers and add extra functionality
– DVSDK DVTB for multiple DVSDK releases – DVSDK Demos for multiple DVSDK releases – Windows & Linux USB Hosts
Page 22
5) Roadmap
Page 23
Roadmap
• Beta: December 2009– Internal and selected External users are able to setup the system and run
existing Test Plans using ‘high-end’ test equipment.• Open source Git Project available
• Contributions limited to internal and possibly a few external users.
• Installation Guide available.
• Users Guide available.
• GA: March 2010– Users are able to develop new test plans on their own and contribute to
the open source project.• Contributions open to all
• Developers guide available.
• Developers how-to videos available.
• Support for less-expensive test equipment, so it is easier for anybody to run tests on their setup.
Page 24
Backup Slides
Page 25
Steps to create automated Test Plan
1. Select Test Equipment to use (if any)
2. Select BEE & TEE to use
3. Define Test Parameters
4. Write Test Equipment driver (if one doesn’t exist)
5. Write Test Script for selected TEE
6. Add new Test Cases/Test Suite to TestLink project. Each test case identifies:
1. Required Software Assets
2. Required Hardware Assets
3. Test Parameters (DUT, Test Equipment, Application)
4. Test Execution Engine type (i.e. VATF, STAX)
5. Test Script
An easy way to remember the information that goes into the test case definitions is by remembering the word SHAPES
Page 26
DISPATCHER
TMC’s StafProc
RESMGR
TMS/User Request
TEE
1
2
5
43
6
1. TEE Register w/ RESMGR and provides its type and capabilities upon startup. It also registers with the dispatcher and provides xslt or exec file to transform the test data.
2. Test Request send to TMC
3. DISPATCHER query RESMGR for available TEE type w/ required HW_ASSETS (capabilities)
4. RESMGR provides TEE machine name
5. DISPATCHER send test request to TEE machine gotten on step 4.
6. TEE completes job execution and notifies DIPATCHER.
7. DISPATCHER releases TEE
8. TEE unregister w/ RESMGR upon shutdown
7
8
TMC’s Resource Manager
Page 27
Test Management System’s Services
• TestLink Basic Definitions:– Test Project holds together the multiple test artifacts (test cases, test requirements, test plans,
etc.)– Test Case describes a testing task and expected results. Test Cases are the fundamental piece of
TestLink.– Test Suite is a collection of Test Cases. It organizes Test Cases to units. – Test Build is the software to be tested. It is part of the Test Plan.– Test Plan is created when you'd like to execute Test Cases. Test Plans can be made up of the
Test Cases from the current Test Project.
• Main Services provided by the TMS:– Store all Test Artifacts – Provides ways to Add/Edit/Delete Test Artifacts:
Test Cases Test Suites Test Requirements Test Builds Test Plans (Test Cases + Build Tag)
• Test Results– Export/Import Test Plans to/from XML– Trigger test execution (optionally)
Page 28
Test Master Controller’s Services
• Dispatch Jobs to appropriate BEE & TEE
• Translate Test parameters between TMS and BEE/TEE
• Provide TMS-independent test case representation to TEE
• Provide a mechanism by which TEE can save test results in TMS
• Manage Service Providers
Page 29
Service Providers’ Services
• At least there will be two types of service providers:– BEE: Build Execution Engines to build the required software assets.– TEE: Test Execution Engines to run the test cases. The TEE maybe as simple as
a shell script or as complex as VATF.• BEE & TEE are the only service providers that the TMC Dispatcher directly
request services from.• Complex TEE, such as VATF, will typically perform these tasks:
– Initialize HW assets (both Device Under Test and Test Equipment)– Create Test Parameters representation – Run Test Execution Logic
• Load Device Under Test (DUT) w/ appropriate SW assets• Boot DUT• Connect DUT to Test Equipment • Configure DUT and Test Equipment• Check conditions or measure metrics• Determine Pass/Fail
– Save Results & Logs