+ All Categories
Home > Documents > Cloud and Software Lifecycle

Cloud and Software Lifecycle

Date post: 17-Sep-2015
Author: dhiraj
View: 11 times
Download: 0 times
Share this document with a friend
Embed Size (px)
of 36 /36
Politecnico di Milano Cloud testing and support to the software lifecycle Elisabetta Di Nitto
  • Politecnico di Milano

    Cloud testing and support to the software lifecycle

    Elisabetta Di Nitto

  • The software lifecycle and the cloud

    ! PaaS/IaaS approaches seen so far mostly dedicated to offer run-time features

    ! Google App Engine and MS Azure also offer programming features

    ! IDEs ! Programming models (e.g., web role/worker role in

    Azure, parallelism managed through task queues in GAE)

    ! Coverage of other aspects of the software lifecycle ! Cloud testing and testing Cloud applications ! CloudBees


  • Software testing

    ! Software testing is the process of assessing, through execution, the functional and non-functional characteristics of a program

    ! For large systems: ! High number of test cases (thousands) ! Need to execute the same tests several times during

    the day ! Need to handle testing for multiple configurations (e.g.,

    multiple browsers, various operating systems, various versions/types of databases, )

  • Software testing and the Cloud

    Cloud characteristics Testing problems Possibility to have a large number of parallel executions

    Large test suites

    Possibility to exploit resources (and pay for them) only when needed

    Execute tests along the day

    Possibility to build virtual machines each one with a different configuration

    Handle different configurations

  • Cloud testing

    ! Cloud testing means using the Cloud to support the testing phase

    ! It seems to be the silver bullet, but it requires care!

  • Risks of Cloud testing

    ! Testing artifacts need to be migrated to the cloud even though still kept in synch with other development activities

    ! Artifacts to be migrated ! Test cases/test suites ! Scaffolding ! Software to be tested ! Testing tools

    ! Specific Cloud issues, e.g., security, should be taken into account

    ! Test code, libraries, dependencies may not be supported by the selected Cloud

  • Characteristics of applications that can be tested in the Cloud

    ! Test case independence ! If test cases are temporally dependent from each other,

    they cannot be run in parallel ! Standard operating environment

    ! Otherwise incompatibilities may arise ! Availability of programmatic interfaces

    ! GUI-based testing could be less convenient on the Cloud

  • Types of testing suitable for the Cloud

    ! Unit testing ! Easy to automatize ! A killer application of Cloud testing

    ! High Volume Automated Test ! Looks for failures that happen only after a prolonged test

    (e.g., deadlock, race conditions, ) ! Resource intensive ! It can rely on the large number of resources offered by the

    Cloud ! Performance testing

    ! Used to determine application throughput, resolving concurrency issues, tuning server response time, assessing availability

    ! In a Cloud environment it is possible to emulate load on demand

  • What do we install on the Cloud?

    ! The entire system to be tested, together with all it is needed for performing testing This is what we have assumed so far

    ! We use the Cloud for hosting/emulating part of the system

  • Examples of testing tool: D-Cloud

    ! Support execution of tests with fault injection machine using the fault injection facility implemented inthe virtual machine layer.

    The computing resources can be managed flexibly. Ifresources are available, then test cases can be executedquickly by simultaneously using the resources.

    D-Cloud automates testing using descriptions of the sys-tem configuration and the test scenario to execute testson cloud computing systems.

    A. Fault injection in a virtual machineD-Cloud uses a virtual machine to execute system tests.

    The virtualized hardware device can simulate failures on theguest OS, and fault injection using virtual machines allowssystem tests to be executed without changing the program.Using the virtual machine, D-Cloud can test software runningin not only the userland layer but also in the kernel layer.When software bugs on the kernel layer are detected duringa system test, the OS may hang-up automatically due to akernel panic. When the system runs on real machine, it isdifficult to obtain helpful information for the bug fix in thiscase, because the user cannot manipulate the OS under thekernel panic. However, when using a virtual machine, a bug inthe OS running on the virtual machine does not affect the hostOS running on a physical machine. Therefore, the tester cancontinue system tests, and the tester can collect informationfor debugging even if the guest OS crashes. Furthermore, thesnapshot of the previous state in the guest OS permits theoperation to return until the desired state repeatedly.

    B. Management of computing resourcesFor developing reliable systems, it is important that system

    tests must be executed for as many cases as possible in orderto find and fix as many bugs as possible. In addition, in orderto execute many tests, a large amount of resources must bemanaged efficiently and flexibly. In D-Cloud, resources aremanaged by a cloud computing system. For example, a numberof systems that require high reliability and dependabilityconsist of multiple nodes linked by a network. In this case,D-Cloud can test such distributed systems using several guestOSes.

    C. Automating system configuration and testingD-Cloud automates the system setup process of the tested

    system and the test process, including the fault injection,based on a scenario written by a tester. When the testerwrites a number of configurations of system test environmentsin a scenario description file, D-Cloud sets up appropriatetest environments and executes appropriate tests automatically.Therefore, D-Cloud enables the behavior of dependabilityfunctions on the system to be tested exhaustively and enablessystem tests to be executed quickly.

    III. DESIGN OF D-CLOUDD-Cloud uses QEMU as virtualization software and Euca-

    lyptus, which is an open-source implementation having thesame API as AmazonEC2[3]. Figure 1 shows an overview ofD-Cloud. D-Cloud consists of the following components:

    Fig. 1. Configuration of D-Cloud

    1) QEMU nodes, which are available with the fault injec-tion facility.

    2) The controller node, which manages the QEMU nodesusing Eucalyptus.

    3) D-Cloud frontend, which issues test and fault injectioncommands and transfers input/output data with QEMUnodes.

    A. Fault injection using QEMU

    D-Cloud uses QEMU as a virtualization software. Theadvantages of using QEMU are described following.

    QEMU can emulate a number of hardware devices. Thus,QEMU may control several hardware faults in the guestOS.

    The QEMU source codes are available as open-sourcesoftware. This allows modification to the hardware emu-lation codes in order to add the fault injection function.

    QEMU can isolate the host OS from the guest OS. Inaddition, QEMU may insulate the computing host fromanomalous behavior of the guest OS during various tests.

    1) Type of fault injection: Table I lists the types of faultinjection . Here, we implement the fault injection mechanismfor the hard disk controller, the network controller, and thememory as target devices.

    2) Reproducibility in fault injection: Reproducibility isimportant in fault tolerant testing using fault injection. If atesting environment has reproducibility, then the tester canfind errors that occur under specific conditions and can fixsuch bugs. In particular, in many cases of hardware deviceerrors, we cannot predict the effects of bugs, and these effectsare usually not repeatable. However, since hardware errors canbe emulated anytime and anywhere using a virtual machine,the costs associated with using a virtual machine are lowerthan those associated with using physical machines.



    device contents valueHard disk Error of specified sector badblock

    Specified sector is read-only readonlyError detection by ECC ecc

    Received data contains error corruptResponse of disk becomes slow slow

    Network 1bit error of packet 1bit2bit error of packet 2bit

    Error detection by CRC crcPacket loss loss

    NIC is not responding nicMemory Bit error bit

    Byte at specified address contains error byte

    B. Managing resources using EucalyptusD-Cloud manages virtual machine resources using Euca-

    lyptus. Eucalyptus is a cloud computing infrastructure thatmanages machine resources flexibly using a virtual machine.D-Cloud consists of multiple QEMU nodes, which executeguest OSes, and a controller node, which controls all of theguest OSes.The procedures to manage machine resources are shown as

    follow.1) A tester uploads OS images to the controller node and

    registers the machine images to the D-Cloud frontend.2) The controller node transfers OS images to QEMU

    nodes.3) The OS images are booted as a guest OS on QEMU

    nodes.After the controller node receives the request to boot a guest

    OS, the controller node transfers the OS images to QEMUnodes, which are available to run the OS images. Thus, thetester does not need to be aware of computing resources onD-Cloud.C. D-Cloud frontendD-Cloud frontend manages guest OSes, configures system

    test environments, and transfers various data from the testerto a guest OS executed for the purpose of system testing.D-Cloud frontend performs its function as follows:1) D-Cloud frontend receives a test scenario, a test pro-

    gram, input data, and an execution script from a tester.2) D-Cloud frontend then issues a request to boot a guest

    OS to the controller node.3) D-Cloud frontend then transfers the test program, the

    input data, and the execution script to the guest OS.4) D-Cloud frontend then issues the fault injection com-

    mand to the target guest OS.5) Finally, D-Cloud frontend collects the output data, logs,

    and snapshots.Since D-Cloud frontend collects data obtained in the test,

    the tester can download these data anytime. If the tester checksthe output and traces the operation using saved snapshots, thetester can discover some bugs and investigate these bugs indetail.

    Fig. 2. Example XML scenario


    D-Cloud executes system tests according to a scenariowritten in XML. By describing the scenario, the tester si-multaneously executes various system tests. Figure 2 showsa complete example of a scenario statement. The scenariostatement consists of four parts defining the test as follows:

    Description for hardware environment Description for software environment Definition of faults for injection Procedure of the entire test

    A. Configuration for the hardware environmentThe definition of the hardware environment is given in

    the machineDefinition element. Table II lists the contentsof the machineDefinition element. Next, the definition ofthe hardware environment is described in the following. ThemachineDefinition element can hold multiple machine el-ements. The machine element includes five elements (name,cpu, mem, nic, and id), and these elements are needed for eachdefinition of a hardware environment. The name element


  • Using the Cloud for emulating part of the system

    ! Emulation of large traffic in stress and performance testing

    ! Application-Behavior Virtualization to replace expensive/limited software

  • SOASTA CloudTest

    ! Use the Cloud to inject large traffic hitting the application under test

    ! Testing can happen also in production

  • Application-Behavior Virtualization

    ! Focuses on enabling testing when the software under test includes

    ! Missing/unstable components ! Non fully accessible third-party/partner systems and

    services ! Systems that are too complex for test labs

    ! Enables testers to rapidly isolate and virtualize just the behavior of the specific dependent components that they need to exercise in order to complete their end-to-end transactions.

  • Application-Behavior Virtualization

    ! Steps ! Identify the components to virtualize ! Capture the behavior of the associated transactions,

    messages, services in virtual assets ! Configure the virtual assets parameterizing conditional

    behavior, performance criteria, test data ! Use the virtual assets to emulate the actual behavior of

    components even when they are not available

  • Test environments in the Cloud (1)

    24 2011 Computer Sciences Corporation

    Test Environment in CloudAttributes of Cloud

    solutionsCharacteristics Benefits

    Advanced virtualization Test resources (infra, tools and people) are pooled and virtualized

    Providing efficient implementation of independent infrastructure

    Rapid Provisioning Test resources are provisioned on demand

    Reducing test setup and execution time and eliminating errors

    Service Catalog ordering Test environment are readily available

    Enabling visibility, control and automation

    Elastic scaling Test environment can be scaled up or down by large factor as the need emerges

    Optimizes, infra and software license usage

    Flexible pricing Test resources are priced on supported topology and project phases

    Offering pricing options tailored to user resource need

    Metering and billing Test resources used in reserved are charged back to LOBs

    Prioritizing innovative projects

  • Test environments in the Cloud (2)

    25 2011 Computer Sciences Corporation

    Test Environment in Cloud (Contd..)

    Attributes of cloud solutions

    Characteristics Benefits

    Re-platform ability (Compatibility Testing)

    Platform based configurations (for testing) can be created and managed

    Comprehensive assurance on product / service

    Maintenance of multiple test beds based for multiple release testing

    Release based configurations (for testing) can be created and managed

    Complete assurance on maintenance of product / service

    Test factories and TCoE Setups Emerging alternative models for supporting client testing needs

    Collaborative platform enables collective productivity

    Service virtualization In a multi component architecture, availability of a dependant component(s) managed for testing

    Effective completion of component level testing, despite the dependencies on Critical components

    ALM (Application Lifecycle Management) / TLM (Test Lifecycle Management) support

    In a multi component architecture, availability of a dependant component(s) managed

    Effective completion of component level testing, despite the dependencies on Critical components

  • Test environments in the Cloud (3)

    26 2011 Computer Sciences Corporation

    Past Present and future (Test Environment)Capability Past Present and future

    Server / Storage utilization 10-20% 70-90%

    Cost Reduction Nil 20-30%

    Self Service None Unlimited

    Test Provisioning Weeks Minutes

    Change Management Months Days / Hours

    Release Management Weeks Minutes

    Metering / Billing Fixed cost model Granular

    Re-platform ability (Compatibility Testing) Prohibitively expensive

    Engineering possibility with affordable cost

    Maintenance of multiple test best beds for multiple release testing

    Prohibitively expensive

    Engineering possibility with affordable cost

    Multiple Tools testing (Application Security for false positives false negatives triangulations)

    Prohibitively expensive

    Affordable because of utility pricing and improved coverage

    New Alternate for BCP (Business Continuity Plan) and DR (Disaster Recovery)

    Not Applicable A Distinct possibility

    Test factories and TCoE Setup for clients 8-12 months 3-6 months

  • Testing as a Service (TaaS)

    ! Potential benefits ! Tool license and infrastructure costs on a pay-per-use

    basis ! Flexibility and wide range ! No setup and procurement time wastage ! No need of hiring expert people

  • The landscape of current offering [CSC]

    30 2011 Computer Sciences Corporation

    Cloud Based Testing Service Offerings under TaaS Ecosystem

    Platforms Windows (Multilingual, Multi

    CPUs) Linux (Red hat, Linux, Debian)

    MS-SQL. Oracle, DB2, Informix, MySQL, Teradata Sun Solaris

    Elastic servers (Tomcat, My

    SQL combination s)

    Borland ES, Cold fusion, Tomcat, Web logic, Web

    Sphere, Web Application server (SAP), ATG Dynamo, JBOSS

    Tools Sandbox

    Apps& Services Client / Server

    Enterprise Applications SAP, Oracle,

    Cadence, PTC, Sugar CRM


    Powered by

    Web Application Performance


    Web Applications

    Industrialized Solution CAT / SAPT



  • TaaS experience at CSC

    33 2011 Computer Sciences Corporation

    CSC Client

    Test Requirements & ApplicationComponents

    CSC On-Demand Testing in Cloud

    FTP / URL

    Virtual Test lab using SkyTap

    Database servers

    Web ServerApplication Servers

    Application Under Test

    Automated Software Quality Tools

    Infrastructure Stabilization

    Test Strategy & Planning

    Test Execution Test Summary Reports

    Certify Application

    Customer requests Testing from CSC

    Online portal / Telesales

    Provisioning / Scheduling of

    Tests CSC conducts

    Testing CSC publishes Test Results &



    Customer Sign-off

    Cloud Orchestration



    CSC & Client

    SOASTA Cloud

    Typical Engagement Model


  • What things are the same about testing cloud solutions?

    From Paul Selway and Scott Rostal, Testing in the Cloud, http://www.tcqaa.org/documents/Testing in the Cloud.ppt C Requirements coverage based testing C Risk-based structured test approach C Test Plan / Test Phases C Test Cases / Test Data / Test Automation C Defect Management / Functional Test C Use of off-shore resources / Virtualization C Politics {Quality / Schedule / Resources}

  • What is different about testing cloud solutions? From Paul Selway and Scott Rostal, Testing in the Cloud, http://www.tcqaa.org/documents/Testing in the Cloud.ppt D Shared multi-tenant test environments D Security (Test User IDs / SSO) D Integration of on/off premise systems D Performance / volume test D Defect isolation D Documentation of Out Of the Box reqs D Release Management

  • What is harder than expected?

    ! Isolating the architectural components for a performance test

    ! Understanding that a cloud solution vendor delivers new releases which you have to take!

    ! Accepting that you have a black box in your environment ! Determining who owns defects

  • What is easier than expected?

    ! Overlapping Test phases ! End User Adoption ! Use of off-shored test resources

    ! Functional ! Performance

  • Enhancements to testing methodology for cloud testing ! Test planning ! Test design ! Cloud setup

    ! Requires familiarity of the target Cloud environment and of its key functions

    ! Understanding of Cloud monitoring tools is essential ! Test execution

    ! Requires usage of additional resources (computational units, storage, communication channels), possibly TaaS

    ! Remediation ! Closure

    ! All used resources need to be released ! Storage resources could be used for archiving test cases,

    results, used VMs could be saved as images and for later use

  • Testing focuses when migrating an application in the Cloud

    ! Test application changes needed to adapt the application to the Cloud

    ! Check on-premise interfaces ! Check data migration and backup mechanisms ! Focus on new security aspects that may have added

    ! Encrypted data in cloud storage ! New secure channels

    ! Performance and scalability ! Compare with on-premise setting ! Make sure that you are making good use of the

    features offered by the Cloud ! Availability and disaster recovery


  • CloudBees main characteristics (1)

    ! Differently from GAE (and Azure), relies on well-known, state of the art components

    ! Tomcat ! Jboss 6 Application Server ! MySQL

    ! Can execute any standard Java web application ! An application built for CloudBees can be moved to any other

    environment, provided that you install the state of the art components it uses

    ! No vendor lock-in ! Offers a load balancer and manages an application cluster

    ! The developer, however, is in charge of maintaining HTTP session on some repository

  • CloudBees main characteristics (2)

    ! Supports a continuous integration process through an on-demand Jenkins (Hudson) service

    ! Each time a new version of a software component is released, the system is automatically built and verified against an array of tests

    ! The build process can take into account the definition of dependences in a Maven repository

  • CloudBees partners eco-system

    ! Examples ! New Relic: performance monitoring service ! Papertrail: aggregation and analysis of log files ! Sauce Labs (Selenium): automated in-browsed testing

    of web applications ! MongoHQ (MongoDB): noSql database ! SendGrid: SMTP service

  • References

    ! Cloud testing ! Parveen, T.; Tilley, S.; , "When to Migrate Software Testing to the

    Cloud?," Software Testing, Verification, and Validation Workshops (ICSTW), pp.424-427, 6-10 April 2010

    ! Banzai, Takayuki et al., "D-Cloud: Design of a Software Testing Environment for Reliable Distributed Systems Using Cloud Computing Technology," Cluster, Cloud and Grid Computing (CCGrid), 2010 10th IEEE/ACM International Conference on , pp.631-636, 17-20 May 2010

    ! SOASTA website http://www.soasta.com/ ! Lef Grant Briefing, H.KOTHANDARAMAN, Testing applications n

    Cloud, http://assets1.csc.com/lef/downloads/LEFBriefing_TestingApplicationsCloud_021011.pdf

    ! WAYNE ARIOLA, The Next Generation of Test Environment Management: Application-Behavior Virtualization, http://virtualization.sys-con.com/node/1905782

  • References

    ! Testing cloud-based applications ! Paul Selway and Scott Rostal, Testing in the Cloud,

    http://www.tcqaa.org/documents/Testing in the Cloud.ppt ! RAJAGOPAL SATTALURI, Testing Considerations for Application Migration

    to Cloud Computing, http://cloudcomputing.sys-con.com/node/1703085 ! CloudBees

    ! http://www.cloudbees.com/