Post on 13-Jan-2017
transcript
Continuous Performance Testing and Continuous Delivery with JenkinsNovember 12, 2015Tom Chavez, SOASTA @TomChavezBrian Dawson, CloudBees @brianvdawson
2© 2015 SOASTA Inc.- All rights reserved.
In This Webinar
TODAY’S PRESENTERS
Tom Chavez: SOASTA, Sr. Product Marketing ManagerBrian Dawson: CloudBees, DevOps Practitioner
TWO COMPANIES, A SINGLE SOLUTION
SOASTA: The leader in Mobile and Cloud Testing
CloudBees: The Enterprise Jenkins Company
Together, we offer a complete cloud-based service:
Continuous Performance Testing & Continuous Delivery with Jenkins
3© 2015 SOASTA Inc.- All rights reserved.
Poll Question
What best describes your status with Continuous Integration / Continuous Delivery (CI/CD)? (single answer)
a)Practicing CI
b)Practicing CD
c)Implementing CI/CD
d)Determining if CI/CD is a good option
e)What is CI/CD?
4© 2015 SOASTA Inc.- All rights reserved.
Agenda
• Why should we performance test earlier? • Building tests and preparing the test environment• Connecting your performance tests to your CI (Jenkins)• Establishing and managing to a performance baseline• Execute tests in parallel with CD pipelines built and executed
with Jenkins Workflow• Q&A
Questions: Submit in the question box during event
5© 2015 SOASTA Inc.- All rights reserved.
Why Should We Load Test Earlier?
6© 2015 SOASTA, Inc.-All rights reserved.
Why Should We Load Test Earlier?
7© 2015 SOASTA, Inc.-All rights reserved.
Why Should We Load Test Earlier?
Okay. Why DON’T we load test earlier?
Copyright © 2014 SOASTA, Inc. - All rights reserved.
8© 2015 SOASTA Inc.- All rights reserved.
Why DON’T We Load Test Earlier?
“Because it’s hard.”o Tools require coding skills we don’t have (C, VB?)o We don’t have a scalable test environmento We don’t have hardware for load generationo It’s a different toolo There are too many other tests to do
9© 2015 SOASTA Inc.- All rights reserved.
Why DON’T We Load Test Earlier?
“Because it’s hard.”o Tools require coding skills we don’t have (C, VB?)o We don’t have a scalable test environmento We don’t have hardware for load generationo It’s a different toolo There are too many other tests to do
“Because it takes too long.”o Scripting takes forevero Results are slowo Fast setup of environments is impossibleo We’re dependent on IT or other teamso We’re too agile
10© 2015 SOASTA Inc.- All rights reserved.
Why DON’T We Load Test Earlier?
“Because it’s hard.”o Tools require coding skills we don’t have (C, VB?)o We don’t have a scalable test environmento We don’t have hardware for load generationo It’s a different toolo There are too many other tests to do
“Because it takes too long.”o Scripting takes forevero Results are slowo Fast setup of environments is impossibleo We’re dependent on IT or other teamso We’re too agile
“Because no one ever told us to.”o There are no “performance coverage” requirementso There isn’t a performance baseline for referenceo There isn’t a defined performance targeto No one is managing to a performance metric
11© 2015 SOASTA Inc.- All rights reserved.
Why DON’T We Load Test Earlier?
“Because it’s hard.”o Tools require coding skills we don’t have (C, VB?)o We don’t have a scalable test environmento We don’t have hardware for load generationo It’s a different toolo There are too many other tests to do
“Because it takes too long.”o Scripting takes forevero Results are slowo Fast setup of environments is impossibleo We’re dependent on IT or other teamso We’re too agile
“Because no one ever told us to.”o There are no “performance coverage” requirementso There isn’t a performance baseline for referenceo There isn’t a defined performance targeto No one is managing to a performance metric
Until Now.
12© 2015 SOASTA Inc.- All rights reserved.
Poll Question
What do you see holding back a more iterative load testing approach? (check all that apply)
a) Skills gaps
b) Lack of test environment(s)
c) Slow test development
d) Test tools limitations
e) 3rd Party Dependencies
f) Undefined Performance Requirements
g) Lack of Management Push
h) Other
13© 2015 SOASTA Inc.- All rights reserved.
Load Testing Has Been an Elite Event
SThe Issue:
• High expectations from tests run with diminishing time windows
• Code-level performance bugs found late set entire project back
• Separate team and schedule conflict with delivery goals
• Major issues escape to production
o pecialty skills and tasks are a legacy barrier
Iteration 1
Automated unit tests
Iteration 2
Automated unit & functional tests
Iteration 3
Regression test suite
Iteration n
Regression test suite
Load & Performance Testing
Deploy
Issues?
14© 2015 SOASTA Inc.- All rights reserved.
Continuous Load TestingValidate performance piece by piece
Iteration 1Automated unit
tests
Deploy
Production perf tests
Iteration 2Automated unit & functional tests
Iteration 3Regression test
suite Incl:
Iteration nRegression test
suite Incl:
End to End Production Scale Test
Small perf test Small perf testsPerf smoke tests Perf smoke tests
The Solution:• Run incremental performance tests automatically with each iteration• Establish a trend line for performance early• Developers find code and architecture issues early• Performance teams focus on end-to-end infrastructure and user related testing• Minimize production escapes and implement production testing
Development/Test team Performance/Ops teamDevOps
15© 2015 SOASTA Inc.- All rights reserved.
Continuous Performance (for Web & Mobile)The Path to a Performance Culture
I JUST MADE A CODE CHANGE
CloudTest runs load tests via JenkinsDaily (or more frequent) updates keep
teams “performance focused”
Identify bottlenecks & inform teams,
just like unit and functional
tests in Jenkins
Jenkins launches suitable cloud-based test environment
Manage with a new metric!
16© 2015 SOASTA Inc.- All rights reserved.
Poll Question
What types of Load Testing are you doing today? (Single Answer)
a) Non-Web Protocols (not mobile)
b) Primarily web protocols
c) Primarily mobile web traffic
d) Both web and mobile
e) All web, mobile & non-web protocols
17© 2015 SOASTA, Inc. All rights reserved.
Enabling Continuous Performance Testing with a Jenkins CI/CD Pipeline
Brian DawsonTwitter: @brianvdawson
Email: bdawson@cloudbees.com
18© 2015 SOASTA Inc.- All rights reserved.
Why are we here today?
Fundamentally we are talking about......
18
DeliveringBetter
SoftwareFaster!
Concept-to-Customer
High Quality• Performance• Feature• Function
RapidlyEliminating WasteMarket Innovation
High Value
19© 2015 SOASTA Inc.- All rights reserved.
How Do You Deliver Better Software Faster?
19
Prod
RapidlyRepeatedly
Reliably
ProdDev
20© 2015 SOASTA Inc.- All rights reserved.
How Do You Deliver Better Software Faster?
20
RapidlyRepeatedly
Reliably
ProdDev
21© 2015 SOASTA Inc.- All rights reserved.
How Do You Deliver Better Software Faster?
21
“DevOps” Has Great Promise
“DevOps is a software development method that stresses communication, collaboration and integration between software developers and information technology (IT) professionals.”[1]
[1] http://en.wikipedia.org/wiki/DevOps
DevOps
22© 2015 SOASTA Inc.- All rights reserved.
How Do You Deliver Better Software Faster?
22
“DevOps” Has Great Promise
“DevOps is a software development method that stresses communication, collaboration and integration between software developers and information technology (IT) professionals.”[1]
[1] http://en.wikipedia.org/wiki/DevOps
DevOps
But You Can’t “Do” DevOps. DevOps an outcome.
23© 2015 SOASTA Inc.- All rights reserved.
You Can “Do” Continuous Delivery
The application of automation and orchestration technologies and principles of Lean Manufacturing to the software delivery lifecycle.
23
24© 2015 SOASTA Inc.- All rights reserved.
You Can “Do” Continuous Delivery
24
DevOpsThe “What”
27© 2015 SOASTA Inc.- All rights reserved.27
DevOps
The “How”
The “What”
Continuous Integration
AgileIteration
Continuous Delivery
You Can “Do” Continuous Delivery
SDLC CoverageFeature, Non-Functional, Function, Operational
Integration and A
utomation
28© 2015 SOASTA Inc.- All rights reserved.28
DevOps
The “How”
The “What”
Continuous Integration
AgileIteration
Process&
Practices
Continuous Delivery
CI and Continuous Delivery (CD) become the foundation for a DevOps transformation.
You Can “Do” Continuous Delivery
29© 2015 SOASTA Inc.- All rights reserved.
!Robin Thicke “Blurred Lines”
“Today, meaningful integration happens less in source control and more frequently in a runtime environment… if you want to quickly verify that the work of two developers is not
conflicting, you need to run API tests, as well as some functional tests scenarios. That means
bringing the various services into an integration test environment, and validating runtime functionality at a pace similar to the classic CI build loop ”
— Steve Smith 2014
Continuous Integration and todays technologies...
The line between Continuous Integration and Continuous Delivery has blurred
30© 2015 SOASTA Inc.- All rights reserved.
CI, CD and Feedback
Did the
Build
break?
Is it fast?
Does it
deploy?Is the “need” met?
31© 2015 SOASTA Inc.- All rights reserved.31
So, How do you “Do” Continuous Delivery?
32© 2015 SOASTA Inc.- All rights reserved.
Automation is the Key
32Photo courtesy of Steve Jurvetson via Flickr
33© 2015 SOASTA Inc.- All rights reserved.
Continuous Delivery (CD) Applies Automation to the Application Lifecycle
ProdDev BuildCommit Test Stage Deploy
Feedback Loop
34© 2015 SOASTA Inc.- All rights reserved.
CD Starts with Continuous Integration (CI): Automating Application Build and Test
Continuous Integration
ProdDev
Automation Engine
Developer
Source Code
Control System
Compile/BuildCode
Scan TestsCodeCommit
Results Report
BuildCommit Test
37© 2015 SOASTA Inc.- All rights reserved.
Jenkins is the #1 Automation Platform Used for CI
Continuous Integration
Developer
Source Code
Control System
Compile/BuildCode
Scan TestsCodeCommit
Results Report
BuildCommit Test
38© 2015 SOASTA Inc.- All rights reserved.
Jenkins is Also Your Key to Continuous Delivery
Developer
Source Code
Control System
Compile/BuildCode
Scan TestsCodeCommit
Results Report
BuildCommit Test Stage Deploy
Deploy Prod
Deploy Stage
Complex Delivery Pipelines
Delivery of App and Config
Continuous Delivery
ProdDev
39© 2015 SOASTA Inc.- All rights reserved.
Have you met Jenkins?
#1 CI/CD Automation Engine
• Open Source
• Highly Extensible
About 8 years old
Easy to install/use
1000+ plugins
Widely adopted
• 100K+ installations
40© 2015 SOASTA Inc.- All rights reserved.
Over 1000 Jenkins Plugins for Tool Integration
Integrated Development Environments (IDEs), editors/compilers/debuggers, code review and collaboration tools.
Source code control systems and related asset management tools.
Automated API-based testing frameworks.
Release management and release automation software
Automated API-based testing frameworks, static code analysis tools.
Load Testing Tools and Security Scanning technology
Automated and manual UAT tools
Release management, release automation, and change control software.
Environment and Configuration Management SoftwareDevelop
Commit
Build & Integrate
Unit Tests
Release(to Test)
AutomatedRegression Tests
Release(to Prod)
User Acceptance Tests
Load & Security Tests
41© 2015 SOASTA Inc.- All rights reserved.
Jenkins is the Hub of the CD Ecosystem
On-premise or in the cloud
Plug-ins for all your tools
RunDeployStageTestBuildCommit
42© 2015 SOASTA Inc.- All rights reserved.
Are You Using Today?
42
Chances are someone in your organization is.
43© 2015 SOASTA Inc.- All rights reserved.43
44© 2015 SOASTA Inc.- All rights reserved.44
How Do You Manage Jenkins and Testing at Enterprise Scale?
45© 2015 SOASTA Inc.- All rights reserved.
About CloudBeesJenkins founder, Kohsuke Kawaguchi, CloudBees CTOWe optimize Jenkins for the enterprise
• Jenkins Operations Center
• Jenkins OSS Support
• Enterprise-class plugins
• Commit code and support the Jenkins Community
• Jenkins training and consulting
• Jenkins User Conferences (JUC) and newsletters
…we are the Jenkins Enterprise experts!
46© 2015 SOASTA Inc.- All rights reserved.
CloudBees Offerings
47© 2015 SOASTA Inc.- All rights reserved.
CloudBees Jenkins Enterprise
47
Professional support
High availability
Large installations
Security
Optimized utilization
Continuous Delivery
48© 2015 SOASTA Inc.- All rights reserved.
How Can CloudBees Jenkins Enterprise Help with Your CD Strategy?
48
Workflow Stage ViewProvides deep insight into the execution of your workflow stages so you can detect problems and performance issues
Manage complex “level-of-detail” testing pipelines
Jenkins WorkflowCreated by CloudBees to provide Jenkins with a way to define flexible delivery pipelines that allow for a rich set of new delivery scenarios
Manage and share “testing-as-code”
Dynamic Environment CreationSpin up new test/deployment environments automatically as part of your delivery pipeline
Deployment Tracking and VerfiicationTrack the history of your deployments – including who kicked off a deployment and what code changes were included in it
Create pipelines that leverage verification tests to determine the quality of a new deployment
Jenkins AnalyticsMonitor and analyze performance across all your Jenkins instances
Monitor, measure and manage adoption of testing, CI and CD best practicesRestart builds, tests, and deployments where they
breakLeverage workflow checkpoints to restart delivery pipelines without a full (expensive) rebuild
...central source of orchestration to bridge dev, test, and ops alike
And more…
49© 2015 SOASTA Inc.- All rights reserved.
Jenkins Workflow and Workflow Stage View Enable Full Continuous Delivery Pipelines
Continuous Delivery
ProdDev
Developer
Source Code
Control System
CodeCommit
50© 2015 SOASTA Inc.- All rights reserved.50
CloudBees Jenkins Workflow Stage View
Restartable Checkpoints
Artifact Downloads
51© 2015 SOASTA Inc.- All rights reserved.
Monitoring, Visualization and Analytics
Jenkins Workflow Stage View Visibility into Pipeline status
Jenkins AnalyticsCross-Team Build Analytics
… among many other features, tailored for enterprise organizations using Jenkins @ scale
52© 2015 SOASTA Inc.- All rights reserved.
Version control
systems
Build & CI Systems
SysAdminsDevOps
Test @ Scale with w/Docker Containers and Cloud Support
Docker Hub
Docker Engine
Developer
$ docker
Benefits• Frictionless movement
from dev > test• Dev + Ops
collaboration in Hub
Test, Stage or Production
53© 2015 SOASTA Inc.- All rights reserved.
Additional Jenkins Services from CloudBees
53
Jenkins Certified Training• Delivered via CloudBees partners
Jenkins Consulting• Delivered via CloudBees Professional Services Team
Jenkins Newsletter• Community news and expert tips
Jenkins User Conferences• Learn from other community members
54© 2015 SOASTA Inc.- All rights reserved.
Continuous Performance Testing
o Demo:
• JBoss Ticket Monster Overview
• SOASTA Cloud Test: Setting up a load test up for unattended runs
• SOASTA TouchTest: Setting up a mobile functional test for unattended runs
• Setting up Continuous Performance Testing
• Jenkins Test Data:
• Reviewing Performance Trending Data: Track performance of each build
• Reviewing Performance Pass/Fail Data: Failing
• Reviewing Functional Test Resultsat are we going to show
What are we going to show
55© 2015 SOASTA Inc.- All rights reserved.
The SOASTA Platform
Performance Analytics
A Continuous Approach: End-to-End Digital Performance Management
56© 2015 SOASTA Inc.- All rights reserved.
SOASTA Jenkins Plug-in
o Open Source: online at jenkins-ci.org
o Automatically available on CloudBees DEV@cloud
o Complete control over CloudTest/TouchTest Automation from within your Jenkins builds:
o Totally integrated into Jenkins environment
Questions?Ask in Question window
58© 2015 SOASTA Inc.- All rights reserved.
Resources
Contact SOASTA:www.soasta.com/cloudtest/info@soasta.com866.344.8766Follow us:
twitter.com/SoastaIncfacebook.com/cloudtest
Knowledge Center• White Papers• Webinar Recordings• Case StudiesCloudLink Community• Support• Tutorials• Video
Contact CloudBeeswww.cloudbees.cominfo@cloudbees.com1.781.404.5100Follow us:
twitter.com/cloudbeesfacebook.com/CloudBees
www.SOASTA.com www.CloudBees.comTry it @ developer.cloudbees.com
Documentation• CloudBees Jenkins Platform• Workflow in Jenkins
White Papers:• Making the Leap from Continuous Integration to Contin
uous Delivery• The Business Value of Continuous Delivery
Continuous Performance Testing and Continuous Delivery with JenkinsNovember 12, 2015Tom Chavez, SOASTA @TomChavezBrian Dawson, CloudBees