Date post: | 11-Jan-2017 |
Category: |
Software |
Upload: | karim-fanadka |
View: | 270 times |
Download: | 0 times |
Continuous Testing2016Karim Fanadka, DevTest Manager
About me: Karim Fanadka
Job: DevTest Manager at HPE Software, 7+
Product: StormRunner Load
Favorite activates: Traveling, Ski, Blogging
Ski – France 2013
Road Trip – Iceland 2015
We are going to talk about:
AgileModern QA
SaaS
Automation
Environments
Hotfixes Continuous Deployment
SaaS – Software as a Service
Before SaaS – On Premise
Software
Users
SaaS – Software as a Service
Data Base
Servers
File System
Firewall
Users SaaS
Which product my team is testing :
StormRunner Load - SaaS product
Performance testing solution that makes it easy to plan, run, and scale web and mobile testing over the cloud.
9
1 Create and run a load test using your favorite scripting tool
2 Load Generators spin up in AWS EC2 or MS Azure
3 Results and app insights are available in real time
The product my team is testing: StormRunner Load
New
Soon
Performance testing solution that makes it easy to plan, run, and scale web and mobile testing over the cloud.
StormRunner Load as a Service
Data Base
Servers
File System
Firewall
Users SaaS Cloud
Agile – Do It Fast
Before Agile - Delivery Timeline
1 or 2 Major releases to production in 1 year
Agile StormRunner Load Delivery Timeline
March April AugustJune Nov Dec
Whats the CodeName of the next release?If you know the answer, I will invite you
to a AlleyCat BubbleBobble Elevator Action Frogger Golden Axe Hang-On
FebruaryDec
CommanderKeen Digger
6 Major releases to production in 1 yearYear: 2015
Agile – The Challenges• Test new features
• Test Regression
• Automation
• Environments
• Continues Integration / Continues Delivery
• Hotfixes
Do it in 1.5 months
• Data Migration (When update SaaS with the newer version)
• Performance Testing
• Security Testing
• Globalization
• Documentation
• And more …
Modern QA Team (DevTest)
What is DevTest? Using top end QA/DEV practices combined with DevOps techniques
QA & DEV Working very close, on the same IDE, code base, and in united process
Automation is a MUST
Manual is for new features / exception flows that are difficult to automate
NO MORE “Deliveries for QA” YES for “Push to Production”
Training“we are testing, and breaking code constantly but we help rebuild it better…….”
Continues Testing
Push new code
Master
Release branch (1.5 months)
Test new codeShort Regression
Push to production
Automation
X Many times a day
Developer Engineer
Test Engineer DevTest Team
Builds in 1 Release (1.5 month): 1,400• Full: 236• Commit: 1,164
Modern QA
DevTest
DevOpsSaaS
Dev
Product
Customer
Automation
Automation
AlleyC
at
Bubble
Comman
der...
Digger
Elevato
r
Frogge
r
Golden
Axe
Hang-O
n
IceClim
ber
JumpM
an0
100
200
300
400
500
600
700
030
80 100 120150 170
200250
300
0 0
75120
147
220
390
450
600650
0 0 0 0
100
200
270 270 270 270
Automation Active Tests - Trend
Client Backend Cloud
85% Coverage
100% Coverage
65% Coverage X 2 Browsers
Builds Number (JumpMan 1.95): 1,617• Number of Full : 427• Number of Commits: 1,190
23
Labels automation coverage SPEC – 66 pages
24
Product Tree – Regression plan
Frameworks:
Frameworks Tests
Client Testing Framework
API Testing FrameworkCloud Framework
Results
28
Bot
29
Testing based analytics from production
API discovery tool
Cost cloud tool
Cloud Testing
Example for Automation challenge17 Regions in the Cloud 15 different metrics
255 Tests
Automation – Cloud Framework - Report
Test Environments
Test Environments Farm
QA Security Performance IntegrationLocal (Linux)
Staging
Hotfixes in 1 day
Hotfixes Lifecycle
07:00Production bug
08:00Research by QA and Dev
10:00Fixing bug
16:00Testing (Staging)
14:00Testing (Local)
18:00 Push to production
Articles • 10 best practices for QA teams to deliver quality software, fast
• Blurring the lines: How the development tester role changes on agile teams
• 3 steps to building test databases for the real world
• Plan your tests and release criteria the right way
Tip
“Changing our mindset takes precedence over technology. When we work together, we will succeed.”
Thank you
@kim0z85
Question?
https://www.facebook.com/karim.f.as
https://il.linkedin.com/in/kareemfa [email protected]@HPE.com
Karim FanadkaDevTest ManagerHPE Software
46
47
12/27/2015 3/6/2016
3/6/2016Go liv e
3/2/2016Staging
2/28/20 16CF
2/14/20 16FF
1/3/2016DevTest Start
Ice Climber V1.9 (Demo T imeline)
• Production
• Staging
• CF
• FF
• Dev
48
CIProductionFFCF StagingSecurity + SAASFeature discovery Defects -> ER’s
In Progress
49
12/27/2015 3/6/2016
3/6/20 16Go liv e
3/2/2016Staging
2/28/20 16CF
2/14/2016FF
1/3/2016DevTest Start
Ice Climber V1.9 (Demo T imeline)
Preparations: Feature discovery (meetings) Defects review (re prioritize, clean) Writing automation spec - example Automation design for exist and new frameworks
Testing: Test new features (Top priority, to make sure no delay in the timeline) Write automation tests to increase coverage (in the first 2 weeks, content is very low)
Feature Freeze
50
12/27/2015 3/6/2016
3/6/20 16Go liv e
3/2/2016Staging
2/28/20 16CF
2/14/2016FF
1/3/2016DevTest Start
Ice Climber V1.9 (Demo T imeline)
10 days for stabilization : Test un finished features (2 days in the FF) Prepare [Cloud deployment, move to Release branch, prepare regression plan, update AGM] Regression ~7 days – Product matrix Migration – Copy DB from production + File system
In addition: Prepare Security Env Demo all the features to Security / SaaS Discover next release features
Code Freeze
51
12/27/2015 3/6/2016
3/6/20 16Go liv e
3/2/2016Staging
2/28/20 16CF
2/14/2016FF
1/3/2016DevTest Start
Ice Climber V1.9 (Demo T imeline)
2 days for final validations : Verify defects that was fixed in the end of FF Validate Security + Performance + Globalization have no holders
Staging ~1 day sanity
Production ~1 day sanity
CI/CD
52
Follow all Jenkins failures Test issue: Fix test Code issue: Contact Dev and fix directly / Open defect CI/CD Env issue: Contact DevOps
Deployment SaaS QA1 / SaaS Q2 / Migration / Security Cloud images
On Premise
Builds Number (JumpMan 1.95): 1,617• Number of Full : 427• Number of Commits: 1,190
Defects
54
Coverage : 65% - example Tests: 700 active tests UI product tree covergae
AutomationFrameworks
Client Backend Coverage: 85% - example API discover
Cloud Coverage: All regions – 20 Regions
Protocols –
Metrics
NV
Save months of WD Cover new LG’s very fast Duration: 1.5 hour
~13 min (all tests)
Commit
Full
~2.5 min (main flows)
Automation
AlleyC
at
Bubble
Comman
der...
Digger
Elevato
r
Frogge
r
Golden
Axe
Hang-O
n
IceClim
ber
JumpM
an0
100
200
300
400
500
600
700
030
80 100 120150 170
200250
300
0 0
75120
147
220
390
450
600650
0 0 0 0
100
200
270 270 270 270
Automation Active Tests - Trend
Client Backend Cloud
85% Coverage
100% Coverage
65% Coverage X 2 Browsers
Builds Number (JumpMan 1.95): 1,617• Number of Full : 427• Number of Commits: 1,190
56
Next slides are reference – DON’T PRESENT
57
Labels automation coverage SPEC – 66 pages
58
Product Tree – Regression plan
59
API Coverage
60
API Coverage