ABN AMRO Transforms with CICD to Accelerate Software Delivery and Improve Security
• DevOn Summit
• Utrecht 14th Mar 2018
• Stefan Simenon
• Stefan Simenon
• Head of IT Tooling & Software Development
• Email: [email protected]
• Tel: + 31 6 51478665
• Studied Physics & Information Technology
• 20+ years IT experience in various roles
• Currently responsible for Tooling, Software Quality & CICD in
ABN AMRO IT department
• Speaker at several software conferences like Jenkins World,
XebiaLabs DevOps Leadership Summit, AllDayDevOps,
Software Quality conferences
Introduction
ABN AMRO is a leading bankwith an operating income of EUR 8588 million
22,000 employees servicing retail, private and corporate finances worldwide
Headquartered in Amsterdam
5,000 associates working in IT
300+ agile teams
ABN AMRO
Many manual handovers and approvals
Long lead time for software delivery
Software quality issues found at a late stage
Code merging happening at a late stage
Inefficient cooperation between DEV and OPS
Big non-frequent releases to Production
Challenges Faced at ABN AMRO
Financial services market is growing fast, on multiple fronts
WaterfallFull Agile enterpriseTraditional Enterprise,
Agile teams
CICD / DevOps Full DevOps enterprise
Agile & DevOps transition
Produce automated builds and detect errors as soon as possible, by integrating and testing all changes on a regular (daily) basis.
High frequency delivery of a tested functional piece of software that can be deployed to production rapidly.
Fully automated process including deployment to production without human interaction.
Continuous Integration Continuous DeploymentContinuous Delivery
The case for faster response to client needs is clear
CICD program: Set-up
Extend techno-logies
Move to ET
Automated production
release
Mature in UT/ST
Start CICD in UT/ST
PAVE THE WAY
ToolingInfra prerequisitesIntegrationPipelines
MAKE IT HAPPEN
Change ManagementMindset & BehaviourSimplify processesCoaching for agile teams
Front end, Java
CICD program: Approach
• CICD is not only about tooling but mainly mindset & behaviour, a changed Way of Working andprocess improvements.
• The project organisation is set up into a cluster with a central and a decentralized orientation.
1. Pave the way: set up the conditions for the teams to get working.
2. Make it happen: the actual ‘decentral’ CI/CD implementation within the teams.
• Agile teams will be supported once the right tools are available, so start with Java/Front End/BPM TIBCO.
• Strong alignment across DEV, OPS and SECURITY departments
• We know other large companies which need 3 - 8 years, and changed their approach along theway.
• Therefore we keep the overall stages in mind, but plan for the coming three months. Focus on learning and improving instead of long term planning.
Pave the Way – Results so far (1)
• All tools required for Continuous Integration implemented and rolled out• Various Continuous Integration pipelines defines and implemented• Pipelines and their integrations are continuously improved and extended
• JIRA agile toolset defined and implemented• Standard Way of Working defined and roll out in progress• From 2000 to 10000+ users in 2,5 years
• Tooling for release and deployment management selected: XL Release and XL Deploy• Release & Deployment management WoW defined and roll out in progress• Test & Production environments for XL Release and XL Deploy delivered, installation process
has been fully automated• Standard CD pipeline for Java/WebSphere, Open Banking and IIB delivered and connected to
standard CI pipeline. • VSTS selected and implemented for applications based on MicroSoft technology• > 100 applications onboarded for automated deployments• > 500 XL Release users
Pave the Way – Results so far (2)
• SonarQube for code quality, HPE Fortify for secure coding, Nexus Life Cycle for OSS library management
• Governance to manage software quality setup and roll out in progress• Build breakers defined and roll out in progress
• Tools implemented to enable automated testing• Test Service Virtualization rolled out• Automated test data management and governance implemented and roll out in progress• Automated Test framework defined and implemented
• Mainframe tools upgraded to latest versions• Identified strategy to clean unused components and activities to recompile programs
based upon latest Cobol compiler 6.1. This will lead to improved memory usage and less MSU usage.
• Mainframe pipeline based on Compuware TOPAZ, ISPW, Jenkins and SonarQube in progress.
Midrange Build & Delivery pipeline: orchestration
Acceptance environment (ET) Production
environment (PRD)
Test environment(ST)
Zero touch platforms
Deployment
Build
Static secure code
Package
Develop
Source code
Build &Unit
Tests
Code quality scans
ContinuousIntegration
Build artefacts
Continuous Delivery
Test data mgmt
ATAF Test suites
Release management
Tooling
Java
Front End
BPM/TIBCO
MicroSoft
Siebel
PowerCentre/
ETLIIB
Mainframe
CoTS
Mobile
Pipelines within ABN AMRO
Dependency scan
Standard CI pipelines within ABN AMRO and build breakers
Check out project from SCM
Developer triggers build
Build project and execute unit tests
Code quality scan
Secure coding scan
PublishDeployable
artifact
N
Y
Build breaker criteria and governance
• Software quality governance in place.
• If software quality criteria are not met, build will fail and software developer needs to fix/improvethe software before being able to publish a deployable artifact.
• Software quality criteria and roll out of build breakers are defined by a development community consisting of central quality teams, representatives in agile teams, our application development partners and security department.
• Initial build breakers in place for software quality, secure coding and dependency management, build breakers criteria will be strengthened in the future.
• Build breakers lead to improved software and less exception discussions in agile teams.
• Senior management commitment in place.
An IT4IT organisation has been set up to enable the CICD implementation
JIRA dedication team
Software Logistics team
Application Deployment
support team
Test tooling team
Application monitoring team
Change & configuration mgmt team
Portfolio mgmtteam
Application logging team
Implement tooling upgrades
Implement new tools
Enhance and improve CICD pipelines
Implement new CICD pipelines
Handle user management
Support Agile teams
Conduct incident & problem management
Mainframe modernization
1.
Automate all repetitive tasks
2.
Integrate quickly and often
3.
Everyone is equally responsible
4.
Keep changes small
5.
Get continuous feedback
ABN AMRO CICD Key Principles
Make It Happen – Results so far
• CICD summer event held incl. CICD leadership program, demo’s, best practice sharing, trainings • Change management program set up with lots of focus upon Mindset & Behaviour
• Various communities set up• Internal meet ups and hackathons regularly held• Platform set up in which teams can present their successes, failures and how they learn• Internal meetups held with external speakers and tooling suppliers (eg. Jez Humble, Josh
Long, Cloudbees, Sonatype, XebiaLabs, SonarSource)
• CICD coaching framework defined and rollout in progress• 100+ boot camps organised and teams coached• Framework based upon certain set of deliverables and team needs• CICD E-Learning module delivered and rolled out
Test environment uptime improved
Improved code quality & secure coding
Improved cooperation across stakeholders
Improved time to market
Improved development processes
Realised benefits within ABN AMRO
Source code mgt
Build & Unit test
Code quality review
PackageDevelop Compo-nent mgt
Deploy Release tests (ET) Deploy
Continuous integration
Continuous delivery
Continuous deployment
Prod checksDeploy Test (ST)
Zero touch platforms
Code push flow Deployment flowBuild, QA and package flow
x3 deployments to UT x2,5 deployments to ET+20% successful Builds -100% Package creation time -75% Testing time
We never thought it would be possible to develop, test
and deploy something completely in one sprint
I-Markets doubled velocity after 1 sprint containing
CICD improvements only
From 4 Internet Banking releases to 18
releases per year
Core review times have been shortened and
violations when merging are being
prevented
Changes are being rolled out as soon as
they are available
Increased velocity
Private Banking Interlnational team reduced
build from 5 hours to 5 minutes
First continuous deployment realised by identity access
mgmt team
Release times halved for teams using XL Release
Take aways
Senior management commitment & involvement
Invest in reducing technical debt
Create a safe environment (failing is ok)
Do not focus on tooling only
Do not underestimate the journey and complexity
Do not focus on long term but small improvements
Database automation
Automate and improve tooling pipelines
Hybrid cloud strategy
Further transform to DevOps
Improve WoW and Mindset & Behaviour
Facilitate increased team autonomy
Way forward
CICD metrics
22
Questions