1
Packaged vs. Custom Application Testing
Shoeb Javed, CTO
September 07, 2017
2
Agenda
• Digital Drivers
• Packaged vs. Custom Apps
• Prioritizing Testing for Packaged
Apps
• Rethinking Automation
• Questions
Presented by Shoeb JavedWorksoft CTO
3
“It is not the strongest of the species
that survives, nor the most intelligent
that survives. It is the one that is the
most adaptable to change”
- Charles Darwin
4
“More than half of the names of companies on
the Fortune 500 have disappeared since the
year 2000.” 1
Disrupt or Be Disrupted!
Don’t Know What Their Industry
will Look Like in 3 Years248%
Feel They May Become
Obsolete in 5 to 10 Years245%
Feel Threatened by Digital
Start-Ups278%
1World Economic Forum2 Research by Vanson Bourne & Dell Technologies
5
26
Enterprise Apps Are Core to Digital Transformation
Chart Source: Mckinsey, http://www.mckinsey.com/business-functions/digital-mckinsey/our-insights/the-case-for-digital-reinvention1 2017 ERP Report, Panorama Consulting
“As postmodern ERP becomes
pervasive, many organizations
will face increasing complexity,
risk and cost that they had not
foreseen and are not equipped
to deal with.”
- Gartner, Transforming ERP to
Postmodern ERP Primer for 2016
Of ERP projects are
viewed as failures.1
Of ERP projects
exceed their budgets.1
But Change is Increasingly Complex
26%
74%
6
Why Do We Need to
Think Differently
About Enterprise
Applications?
7 9/12/2017 © 2017 Worksoft, Inc.
Today’s Modern Enterprise Applications. Houses or Cities?
8
Design (Fit/Gap)
Requirements
SDLC is Different
• Type of work
• Project scale
• Scope of application functionality
• Size of the “foundation” to be laid
• Size of minimal unit of functionality providing business value
• Potential impact of production deployment failures
Build (configuration, master data,
write code; integrate entire business
processes; unit test)
Test
Functional, Regression,
Integration, UAT, Performance
Deploy to Production
Custom
Development
Build (write code, unit
tests)
Developer Test
Deploy
Requirements
Architect
Integrate
UI, backend
TestFunctional, Regression, UAT
Deploy
Packaged
Applications
9
Test Automation Considerations
Understanding Complexity UI Driven
10
Understanding
11
Complexity
12
The UI is Critical.!It’s what the world sees!
What happens
when we don’t!
End-to-End System Tests and User
Acceptance Tests are run at the GUI level.
It’s what the world sees and validates whether or not
the business process actually works.
13
Business Analysts/QA
“Does my Business Work?”
Developers/Test Engineers
“Does my Software Work?”
UI Tests – The Bulk of Packaged App Tests
13
End to End (UI)
Integration ( API)
Unit (Code)
. UI Testing Remains the Most Challenging to
Automate!
14
Mainframe Client Server Web 1.0 Web 2.0 / Mobile HTML 5 VR & Biometrics
1980 2000
1990
2020
2010 2030
Modern Day UI EvolutionAs the way applications are written changes, so must the way we test.
15
Example. What’s Your UI? Fiori. Ok. But Which One?
SAP Fiori Cloud DemoSAPUI5
v 1.26 – 1.28
Image source:
https://demo-
fioritrial.dispatcher.hana.ondemand.com/sites
S/4HANA 1511 SAPUI5 v 1.30.x or higher
(Fiori 1.0)
Image source:
https://experience.sap.com/fiori-design-
web/sap-fiori/#sap-fiori-10
S/4HANA 1610 SAPUI5 v 1.40.x or higher
(Fiori 2.0)
Image source:
https://experience.sap.com/fiori-design-
web/sap-fiori/#sap-fiori-20
16
How Do We Prioritize Test Automation?
17
Prioritization Starts with the Business!
“Quality and testing in today’s fast-moving
continuous delivery environment is quickly
becoming everyone’s responsibility, so
management need to ensure their people are
properly enabled.”
- Freeform Dynamics, Continuous Testing as a Digital Business
Enabler Research Report, 2017
86%Of testing leaders report they
need to work closer with
business leadership!1
1PAC Group, Digital Testing in Europe: Strategies, Challenges, & Measuring Success, 2015
OperationsEasy to Scale
QAEasy to Use
Easy to Maintain
Application
Team (IT)Easy to Integrate
BusinessEasy to Use
18
Late Defect Discovery is Expensive and Disruptive!
0
9599
22781331
$18 ($0)
$180 ($1.7 M)
$360 ($820K)
$1,800 ($2.3 M)
$0
$200
$400
$600
$800
$1,000
$1,200
$1,400
$1,600
$1,800
$2,000
0
2,000
4,000
6,000
8,000
10,000
12,000
Discovered inRequirements/Specs
Discovered in TestExecution
Discovered in UAT Discovered inProduction
Defect Correction Cost (5 Project Sample)
Defects Cost to correct per defect
19
03
02
01
DEPLOY AT AN
INDUSTRIAL SCALE
IN THE CLOUD
FILTER BASED
ON RISK
BUILD AUTOMATION
FASTER
Pre-built content and
optimizations for popular
enterprise applications.
Automatic test generation
from automated discovery.
Intelligent AutomationVelocity/Execution
Distribute automation across multiple
cloud machines (set up/tear down
machines automatically) to achieve
industrial scale - run millions of process
steps several times a day.
Run only the tests necessary
through impact analysis and other
risk factors associated with a
change.
20
To Succeed
We Need to
Rethink
Automation
1 + 1 = 10x
21
Transferable Automation
1 + 1 = 4!
Automated Discovery Used to Capture a
End-to-End Business Process
Operations• Documentation
• Reuse for Robotic
process automation
QA• Documentation
• Test Processes
Application Team (IT)• Documented requirements
• Work sized, resources allocated
• Stories created for sprints
• Integration with ALM & defect tracking
Business• Visualization of process
• Detailed documentation
generated for training, IT, QA
and Ops
Reduction in manual hours
vs. test automation alone30%
22
Automated As-Is Business Process Discovery
23
Automatically Generated Documentation
QA- and Business Analyst-Friendly Screens
24
Enable Agile Delivery
• Leverage your existing quality management processes while leveraging next-generation test automation
• Use process documentation to define requirements to be associated with test processes for traceability coverage reporting
• Schedule and trigger tests directly from ALM/CD pipeline and bring back test results
25
Auto-Generated Test Cases
26
Document
Generation
Data Loading
Configuration
THINK? How Else Can We Apply Automation Across the Business for Better Results?
Configuration
Robotic Process
Automation (RPA)
Document
Generation
27
5,548 1,584 1,584No Analyze
Project/Blueprint Realization Final Prep Go Live/Hypercare
29%Reduction in
total testing
man days.
28%Increase in
focused
testing, yet
pulled in
GoLive.
30%Reduction of
2,174 total
man days for
project.
Transferable Automation Drives Additional Savings
3,960 2,186 396Analyze
9,528
7,354
75% Reduction in
Hypercare
man days.
Realization Final Prep Hypercare Total Acceleration
812
812
28
WorksoftIntelligent Automation for Enterprise/Packaged Application Projects
Process Discovery,
Documentation &
AnalyticsContinuous business insight
for digital transformation through machine learning
Business Process
Testing & RPAAccelerate Innovation,
Mitigate risk & Streamline Operations
29