© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
◉ Testing Trends:
Virtualization, Mobility, Cloud, etc.
Theresa Lanowitz, voke Founder
Denver, CO | 17 September, 2013
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
◉ Agenda
• About voke
• Global Trends
• QA Today
• Extreme Automation
- Service Virtualization
- Development Testing
- Mobility
• Embracing Extreme Automation
• Net/Net
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
๏ About voke
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
◉ About voke
• Independent industry analyst firm
• Founded in 2006 by Theresa Lanowitz
• Firm focuses on the application lifecycle and its transformation
including virtualization, cloud computing, embedded systems,
mobile and device software
• Provides strategic, independent, and impartial advice and market
observations through qualitative and quantitative research
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
๏Global Trends
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
◉ Current Reality
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
◉ NASDAQ – August 22, 2013
A “computer malfunction”
froze thousands of
securities listed on the
Nasdaq Stock Market for
three hours. Many of the
country’s most-traded
shares ground to a
standstill as brokers were
unable to execute orders.
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
◉ Sabre Holdings – September 11, 2012
American Airlines,
Southwest Airlines,
JetBlue Airways, and
Alaska Airlines lost
access to their
reservation systems
hosted by Sabre Holdings, a global distribution system.
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
◉ More Software Failures Daily
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
◉ Why Are Software Failures Making Headlines?
• Economy
- Lack of investment during Global Financial Crisis (GFC)
• Faster, faster, faster
- Faster is not equal to better
• Old technology
- Most organizations are not up to date on current software platforms
- Most organizations are not leveraging the power of modern solutions
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
YouTube reported a peak of more than 8
million concurrent livestreams of Felix
Baumgartner’s 24-mile fall to Earth.
◉ Red Bull Stratos & YouTube – October 14, 2012
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
๏QA Today
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
◉ QA Roadmap – Critical for Business Success
Chaos
Establish
QA Role
Dev and
QA Parity
IT & Biz
Alignment
Change
Agent
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
◉ QA Roadmap – Leadership and Innovation
Relationship
Leadership
Process
Leadership
Technology
Leadership
Professional services
Software vendors
Line of business / customer
advocacy
Center of Excellence (CoE)
Appropriate metrics
Brand assurance and advocacy
mALM
Lifecycle virtualization
Development testing
Continuous integration
Build managment
Requirements
Application performance
Application security
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
◉ Parity – A Basic Requirement
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
๏Extreme Automation
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
Solving classic computing problems across the
lifecycle through the use of modern tooling
technology.
Removing barriers and facilitating
communication, collaboration, and connectivity
of development, QA, and operations to support
the line of business and the insatiable demand
for quality software.
Source: voke – June 2013
◉ Extreme Automation – Definition
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
◉ Classic Challenge – Deliver All Three
Cost
Quality Schedule
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
◉ People, Process, Technology – Building Blocks
People Process Technology
• Internal skills
• Trusted partners
and providers
• Customer advocate
• Relationship
manager
• Process designation
• Communication
• Collaboration
• Optimized
• Testing platform
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
◉ Minimize Risk – Move to Extreme Automation
How Development Ensures
Working Software
Technology
Companies
Enterprise Companies
Continuous Integration 61% 47%
Code reviews 60% 64%
Automated unit tests 63% 38%
Manual unit tests 46% 56%
Automated functional testing 51% 40%
Manual functional testing 59% 64%
Performance testing 58% 42%
Source code analysis tools used 29% 11%
Code refactoring 39% 20%
Database refactoring 20% 9%
Design patterns 21% 16%
Domain-driven design 14% 4%
Source: voke Market SnapshotTM Report: Agile Realities – July 11, 2012
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
◉ Factors Driving Extreme Automation Need
1. Composite applications
2. Software supply chain
3. Consumerization of IT
4. Demand for quality
5. Mobile
6. Cloud
7. Competitive business differentiation through software
Source: voke Market Mover ArrayTM Report: Testing Platforms – June 5, 2012
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
๏Service Virtualization
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
Server Desktop Lifecycle (Service)
Market Adoption
Phase
Mature Trending Evolving
Technology
Phase
Stable and full
featured
Stable with
emerging
features
Stable and
evolving
Primary Users Data center Service
center
Dev, test, and
operations
Vendors VMware,
Microsoft, Citrix
Citrix,
Microsoft,
VMware
CA, Parasoft, HP,
Microsoft, IBM
◉ Evolution of Virtualization
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
Enabling development and test teams to
simulate and model their dependencies of
unavailable or limited services.
Removes constraints and wait times frequently
experienced by development and test teams to
access components, architectures, databases,
mainframes, etc.
Source: voke Market SnapshotTM Report: Service Virtualization – December 2012
◉ Service Virtualization – Definition
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
◉ Service Virtualization – Sample Use Cases
• Environment replication
• Accessibility
• Supply chain availability
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
◉ Access to Required Systems
33 Average number of
systems needed
for dev or test
18 Average number
of systems with
unrestricted
access for dev
or test
Source: voke Market SnapshotTM Report: Service Virtualization – December 2012
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
◉ Lab Access – Wait Time
96% Wait for
access
Wait Time to Access Lab Participant PercentagesNone – immediate access 4%
1 hour or less 7%
2-4 hours 14%
5-8 hours 11%
2-4 days 38%
1 week 11%
2-4 weeks 14%
10 weeks 1%
Source: voke Market SnapshotTM Report: Service Virtualization – December 2012
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
◉ Lab Access – Restrictions
69% Time
restrictions
Lab Access Restrictions Participant Percentages4 hours or less 31%
5-7 hours 6%
1-2 days 22%
1 week 6%
4-6 weeks 4%
No restrictions 31%
Source: voke Market SnapshotTM Report: Service Virtualization – December 2012
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
◉ Service Virtualization – Rationale for Adoption
Rationale for Adoption Participant PercentagesRestricted access to dependent
services, components, or systems72%
Scheduling constraints 62%
Parallel development 59%
More realistic performance testing 33%
Other 7%
Source: voke Market SnapshotTM Report: Service Virtualization – December 2012
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
◉ Service Virtualization – Use
Types of Virtualized Assets Participant PercentagesApplications 65%
Third-party services 56%
Third-party applications 53%
Mainframe 37%
Partner software 34%
Architecture 22%
Other 4%
Source: voke Market SnapshotTM Report: Service Virtualization – December 2012
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
Category Metric
Reduced Wait Times 64% -achieved a 50% – 100% reduction in
wait time
Greater Availability of
Services
75% - achieved a 50% or greater increase in
availability of services
Decreased Software Cycles 55% - achieved a 25% - 50% reduction in
overall software cycle
Increased Test Coverage 59% - achieved 50% - 90% increase in test
coverage
Shorter Test Cycle Times Average 58% decrease in test cycle length
Improved Time-to-Market Average 24% faster time-to-market
Fewer Production Defects Average 22% reduction of production
defects
Source: voke Market SnapshotTM Report: Service Virtualization – December 2012
◉ ROI Service Virtualization – Summary
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
๏Development Testing
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
◉ Development Testing – Definition
Complement traditional blackbox functional testing.
Transform the software testing activity from reactive to proactive by
moving testing upstream to identify and remediate defects earlier in
the lifecycle.
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
◉ Traditional Cost of Rework
Source: voke Strategic Brief: Cost of Rework Models for Agile and Non-Agile Projects– July 2012
Requirements Design Development TestCustomerSupport
Cost of Large Fix $7,500 $32,500 $107,500 $159,000 $365,000
Cost of Medium Fix $1,500 $7,500 $11,500 $18,000 $40,000
Cost of Small Fix $1,000 $1,500 $2,000 $3,500 $7,500
$0
$50,000
$100,000
$150,000
$200,000
$250,000
$300,000
$350,000
$400,000
$450,000
voke Cost of Rework Model
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
◉ Development Testing – Overview
Primary User Tool Types Function Vendors Developers Static code analysis Defect prevention
• Memory leaks
• Deadlocks
• Race conditions
Coverity
Parasoft
Developers Dynamic code analysis Detect dependencies
based on system execution
Coverity
Parasoft
Developers Unit testing Smallest testable part of an
application to ensure code
meets its design and
behaves as intended
Coverity
Microsoft
Parasoft
Developers
QA professionals
Application security testing Identify security
vulnerabilities through
scanning and penetration
testing
Coverity
HP
IBM
Parasoft
Development Managers Policy Management Define consistent coding
standards across teams
Coverity
Parasoft
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
◉ Development Testing – Benefits
• Test at every phase of the application lifecycle
• Collaborate and communicate with development to increase test
coverage
• Remediate defects earlier in the lifecycle to save time and money
• Reduce traditional testing cycle by focusing testing on areas that
require more thorough testing (increase code coverage)
• Reduce the number of defects entering production
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
๏Mobility
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
◉ Evolution of Complexity – Testing Challenges
Mainframe Client/Server Web Mobile
Co
mp
lex
ity
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
◉ Pillars of Testing – Adding Mobile
Does it work? Does it perform? Is it secure?
Functional Performance Security
Lo
ad
S
tress
En
du
ran
ce
Netw
ork
s
Pla
tform
s
…
Re
qu
irem
en
ts
Man
ua
l
Reg
ressio
n
Ex
plo
rato
ry
Netw
ork
s
Pla
tform
s
User E
xp
erie
nce
…
Sc
an
s
Pen
etra
tion
Sta
tic A
naly
sis
De
vic
e
Da
ta
Co
mp
lian
ce …
= Additive for mobile
Infra
stru
ctu
re
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
◉ Modern Quality Challenge – Mobility and You
• Quality
- Mobile app testing IS application testing
- Mobile strategy is a necessity not a luxury
- Mobile apps demand high quality
- Mobile apps are visible
- Mobile apps are linked to the brand
- Supply chain issues are immediate, visible, and carry consequences
• Tools and Technology
- Innovation of mobile testing is tied to the HP testing assets
- Mobile app testing solutions integrated with and extending existing testing
platforms have an advantage
- Mobility will ultimately be a feature of the broader testing platforms
- Lifecycle virtualization solutions enable mobile app testing
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
◉ Mobile Apps – Best Practices
• Mobile testing IS application testing
• Extend enterprise best practices to incorporate mobile
uniqueness
- Gestures
- Location
• Leverage existing skills in QA
- Platform tooling
- Automation
- Performance
- Security
• Include new team members with specific mobile skills
- Hardware
- Infrastructure
- OS
- Networking
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
◉ Mobile App Testing – Vendors
Vendor HP Asset Use Integration
Experitest YES HP, IBM, Microsoft, Open
Source
Keynote DeviceAnywhere YES HP, IBM
Mobile Labs YES HP
Neotys YES HP QTP/UFT – offers mobile
load/performance testing
Parasoft YES HP, IBM
Perfecto Mobile YES HP
SOASTA YES IBM, Microsoft
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
๏ Embracing Extreme Automation
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
◉ Extreme Automation – Benefits
• Goal – Extreme Automation
• Automation is critical
- Traditional applications
- Mobile
• Reduces risk associated with multi-platform delivery
- Mobile demand
• Reduces overall software cycle time
• Increases code coverage
• Assists in managing the dynamic nature of mobility
• Enhances visibility into the software supply chain and reduces
risk
• Enhances communication and collaboration throughout the entire
lifecycle
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
◉ Extreme Automation – Getting There
People Process Technology
Leverage
relationships with
vendors
Select pilot projects
for new technology
Get current and stay
current on existing
tools
Leverage
relationships with
professional service
providers
Form a CoE for new
technology
Evaluate new tools
offered by current
suppliers
Update internal skills
to focus on achieving
extreme automation
Leverage a CoE
model to go from pilot
project to department
/ enterprise use
Explore innovation
from other vendors
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
๏ Net/Net
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
◉ Net/Net
• Software and the brand are linked
• QA organization is a customer advocate
• Embrace people, process, AND technology
• Deliver on extreme automation - Reduce the risk of failure in production
• Service virtualization is real and proven - Time-to-market
- Defects entering production
- Reduced software cycle
- Reduced test cycle time
- Increased test coverage
• Move testing upstream with development testing - Increase code coverage
• Mobile testing IS application testing - Identify YOUR mobile strategy
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
◉ Learn More at www.vokeinc.com
• Become a Research client at www.vokeinc.com
- voke Market Snapshot: Service Virtualization
- voke Category Snapshot: Lifecycle Virtualization
- voke Market Snapshot: Virtual Lab Management
- voke Market Snapshot: Agile Realities
- voke Strategic Brief: Cost of Rework
- voke Market Mover Array: Testing Platforms 2012
- voke Market Mover Array: Application Lifecycle Management
• Follow voke on Twitter: @vokeinc
• Take our surveys
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
Thank You!!!
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
๏ Q&A
© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.
◉ voke Contact
voke, inc.
2248 Meridian Boulevard
Suite H
Minden, NV 89423
P: +1.866.895.9045
W: www.vokeinc.com
T: @vokeinc