27
Mixing Waterfall, Agile and Outsourcing at Dutch Rail Bob Harnisch, ProRail Tim Koomen, Consultant at Dutch Rail www.eurostarconferences.com @esconfs #esconfs
Mixing
Waterfall, Agile and Outsourcing
at Dutch Rail
Bob Harnisch, ProRail
Tim Koomen, Consultant
at Dutch Rail
www.eurostarconferences.com
@esconfs
#esconfs
� Who am I
◦ Bob Harnisch
◦ Manager Testservices at ProRail
ICT Services
ICT Projects
Project
� ProRail
◦ Railway company in The Netherlands
◦ Provides a secure, reliable, punctual and sustainable rail network
◦ Develops and maintains rail traffic control systems for rail traffic controllers
◦ Problem:
� Control systems must Go-Live first-time-right! (99.99x % availability)
� Multiple vendors and suppliers, mixed old and new project approaches
TestservicesProject
managers
Planning
Control
• Real-time control systems• Safe and reliable
High available systems 24/7Control
Messaging
Safety
Infra elements
• High available systems 24/7• Multiple system chains• Legacy and new• Mixed platform (VAX VMS, Linux)
� Testservices◦ policy: formal testing department for IT projects
◦ involved at project start-up
◦ responsible for acceptance testing
◦ knowledgeable and experienced test experts
� Structured test approach tailored to ProRail
Acceptance Test Centre “Modelpost” (close to 100% representative)� Acceptance Test Centre “Modelpost” (close to 100% representative)
� Technical, complex, real-time, high-availability system environments
� Non-functional testing very important
� But also office automation (SAP, Sharepoint, GIS)
� Different project approaches◦ Policy: outsourcing of SW development
◦ Agile/scrum approach
◦ Waterfall approach
� (Many) external SW suppliers◦ Little business knowledge
◦ No representative test environment◦ No representative test environment
� Changing architectures◦ From old legacy to SOA
� Testing is vital for operation◦ Installation window 01:00 – 04:00 hrs weekend night
� 1st time right => next window in 6 months
◦ Any installation or operational failure:
� We’re instantly on the news!
� How to cope with issues …?
� Anyone here that can give me advice……?
� Tim Koomen, independent testconsultant
8
� Baseline of testing is good
� Last few years explosive increase of agile/scrum development at suppliers. Questions:◦ Is separate acceptance testing still necessary?
◦ If so, how to align planning of sprints with acceptance tests at test centre◦ If so, how to align planning of sprints with acceptance tests at test centre
◦ How to coordinate all testing activities?
◦ Controlling the quality of testing?
� Shift to Service Oriented Architectures (ESB, Tibco)◦ Different testing?
◦ Other measures?
9
Gebruikerswensen
Business Beleid Acceptatietest
(Pilot)
Busi-nessCase
Context
VL
(OCD
Post21)
Proces
boek
VL
Proces
boek
VL
- Proces-
sen (SSS
Post21)
-KPI’s
Proces
boek
VL
-Bedrijfs-
objecten
(SSDD
Post21)
-Externeeisen
- wetgeving- e.d.
OCDBeheerProcedures
-Organi-satie en
Geautoma-tiseerdeOnderst.
Project
Brief
LocatieSpecifiekeTest
STD STR
STD STR
STD STR
STD STRPilotPlan
Vrijgave
BusinessRequirements
BusinessOntwerp
MTPsysteem
STPsysteem
MegaIntegratieTest
ValidatietestSysteem
Systeem
Review
Subsysteem Ontwerp
SysteemSpecificatie
Systeem Ontwerp
SubsystemenIntegratie
SSS
(Use
Cases)
IRS
(externe
systemen)
SSDD
IDD
SubsysteemSpecificatie
Systeem
Systeem =>Subsystemen
SRS
IRS
sub
systemen
Subsystemen
Subsystemen
SDD DBDD IDD
STD STR
STD STR
STD STR
STD STR
UnitCoderen &
Test
STPSysteem/
Sub-systemen
Decharge leverancier
Knip
HLOB
IRS
sub
systemen
SysteemKwalificatieTestIntegratie
TestSubsystemen
SubsysteemKwalificatieTest
Review
Legenda:• ProRail architecten,
ontwerpers of andere ProRailonderdelen, in ieder geval buiten het werkgebied van TS
• project, ontwerpers • leverancier• integrator• Test Services
10
Businesspolicy
Businessrequirements
Businessdesign
Acceptancetest
Goinglive
11
Architecturedesign
Scrum developmentand system
test
� Representative test environment
� Testing non-functionals
� Testing the chain of systems:� at ProRail 1st time large scale system integration
� Operational management tests
� Many suppliers, many scrum teams:� Many suppliers, many scrum teams:� quality not always predictable
� Few demands on supplier testing:� mostly deliverables like test plan, design and report
� Suppliers have little business knowledge
� Configuration management of application/environment:� not always as it should
12
Businesspolicy
Businessrequirements
Businessdesign
Goinglive
PoCor
Pilot
Shadow run
Architecturedesign
Scrum developmentand system
test
Acceptancetest
13
Businesspolicy
Businessrequirements
Businessdesign
Goinglive
PoCor
Pilot
Shadow runTypical scrum tests:
Unit testCode review
Typical acceptance tests:
(Service) integration testValidation test
Architecturedesign
Scrum developmentand system
test
Acceptancetest
14
Code reviewFunctional test (user story test)(Service) integration testNon-functional testsRegression test
Validation testChain testUATNon-functional testsInstallation testOperational acc.test
Businesspolicy
Businessrequirements
Businessdesign
Goinglive
PoCor
Pilot
Shadow run
ValidationTest
InstallationTest
Non-functional
Deliverables:• Testplan• Test scripts• Test report
(Service) Integration
Tests
Business architect
cafeteria
15
Architecturedesign
System requirements, user stories
System-design &
specification
Coding
Code reviews
Unit tests
Non-functional
tests
(automated) Regression
test
Functional(qualification)
tests
(Service) integration
tests
Test
Chain test User acceptance
test
functionaltests
Architect / designer
Deliverables:• Project Start Architecture
Product Owner / Supplier/ ProRail DevelopmentTeam
Definition of Done / deliverables:• Test scripts• Test Report• User Manual• Installation Guide
Testservices
Integrator
Architectural changes to SOA
� Measure 1: test effort on: services, non-functionals & platform releases◦ Experience:
� non-functional requirements missing or not well described (architecture documentation )
� (Lack of attention for) non-functionals is a major cause for project delay
Improve quality at beginning of project
� Measure 2: improve review process� Measure 2: improve review process◦ We have initiated an improved review process, together with architects
◦ Review the Product from 4 Directions:
� Up: Norms, standards, reference architecture
� Left: Consistency with other system documentation
� Right: Usable (for users), testable (testers)
� Down: Designable, Buildable, Implementable
◦ Use of roles versus directions
� (Peer) architect reviews architecture document from Up and Left
� Designer/ Tester/ Developer: reviews from Down, Left, and Right
� Sr user reviews from Right
◦ Reviews form includes checklist per role
16
4DR
Businesspolicy
Businessrequirements
Businessdesign
Goinglive
PoCor
Pilot
Shadow run
ValidationTest
InstallationTest
Non-functionaltests
(Service) Integration
Tests
Businessarchitect
4DRcafeteria
Deliverables:• Testplan• Test scripts• Test report
17
Architecturedesign
System requirements, user stories
System-design &
specification
Coding
Code reviews
Unit tests
Non-functional
tests
(automated) Regression
test
Functional(qualification)
tests
(Service) integration
tests
Test
Chain testUser
acceptancetest
tests
Architect /designer
Product Owner / Supplier/ ProRail DevelopmentTeam
Test services
Integrator
4DR4DR
Deliverables:• Project Start Architecture
Definition of Done / deliverables:• Test scripts• Test Report• User Manual• Installation Guide
Businesspolicy
Businessrequirements
Businessdesign
Installation
Goinglive
PoCor
Pilot
Shadow runningBusinessarchitect
Users
Deliverables:Testplan,
4DR
cafetaria
System requirements, user stories
System-design &
specification
Coding
Code reviews
Unit tests
Non-functional
tests
(automated) Regression
test
Functional(qualification)
tests
4DR
(Service) integration
tests
ValidationTest
Chain test
InstallationTest
User acceptance
test
Non-functionaltests
Test services
Architect /designer
Testplan, testscripts, Testreport
Definition of Done / deliverables:Testscripts, testreport, user maual,
installation guide
Legenda:• ProRail architects, designers or other
ProRail roles, outside scope Test services
• project, developers• Supplier, ProRail product owner• Test Services
(Service) Integration
TestsArchitecture
design
4DR
Deliverables:Project Start Architecture
Integrator
Product Owner / Supplier/ ProRail DevelopmentTeam
18
Businesspolicy
Businessrequirements
Businessdesign
Installation
Goinglive
PoCor
Pilot
Shadow runningBusinessarchitect
Users
Master Test PlanMTP
4RR
cafeteria
Deliverables:Testplan,
System requirements, user stories
System-design &
specification
Coding
Code reviews
Unit tests
Non-functional
tests
(automated) Regression test
Functional(qualification)
tests
4RR
(Service) integration tests
ValidationTest
Chain test
InstallationTest
User acceptance
test
Non-functionaltests
Test services
Architect /designer
Legenda:• ProRail architects, designers or
other ProRail roles, outside scope Test services
• project, developers• Supplier, ProRail product owner• Test Services
(Service) Integration
TestsArchitecture
design
4RRIntegrator
Product Owner / Supplier/ ProRail DevelopmentTeam
19
Testplan, testscripts, Testreport
Definition of Done / deliverables:Testscripts, testreport, user maual,
installation guide
Deliverables:Project Start Architecture
� Implement it� Train testmanagers, implement in projects
� Build up experience� 1st Project� 1st Project
� 2nd Project
� ……
� Next Project
� Some time later….
20
• Satisfied with– Roll-out and acceptance of model– Good fit to current practice, clarifies testing– Early identification & alignment of test activities at suppliers and ProRail
gives flexibility in finding test solutions– Early involvement– Coaching of test managers – Largest changes: choice and frequency of tests, tighter control of
Product owner
Stakeholders
Scrum team
– Largest changes: choice and frequency of tests, tighter control of supplier tests and working in or with scrum teams
• To be improved– 4 Direction Reviews, as Testservices is not the natural owner– Influencing and controlling the supplier tests
• Not a test issue– Getting used tot Agile/scrum– Agile / scrum in general: product owner is the (weakest!) link …– … and the hardest role!
21
� Waterfall
� Agile
� Combined
� United
Outsourcing
WAUW� Outsourcing
� Testing
� ProRail
� …
22
Businesspolicy
Businessrequirements
Businessdesign
Goinglive
PoCor
Pilot
Shadow runningBusinessarchitect
Users
Master test planMTP
4RR
cafeteria
Deliverables:Testplan,
testscripts,
System requirements, user stories
System-design &
specification
Coding
Code reviews
Unit tests
Non-functional
tests
(automated) Regression test
Functional(qualification)
tests
4RR
(Service) integration tests
ValidationTest
Chain test
InstallationTest
User acceptance
test
Non-functionaltests
Test services
Architect /designer
Legenda:• ProRail architects, designers or
other ProRail roles, outside scope Test services
• project, developers• Supplier, ProRail product owner• Test Services
(Service) Integration
TestsArchitecture
design
4RRIntegrator
Product Owner / Supplier/ ProRail DevelopmentTeam
23
testscripts, Testreport
Definition of Done / deliverables:Testscripts, testreport, user maual,
installation guide
Deliverables:Project Start Architecture
� ProRail’s test solution for hybrid situations: WAU! - testmodel
� Acceptance testing is a part of this solution
� Flexibility in choosing the required tests� coordinated by a Master Test Plan
Early test involvement in reviews� Early test involvement in reviews� 4DR – 4 Directions Review
� Model for communication and awareness at start project� testers and others in project organization
� In future to 100% agile?
�We’re ready!
24
25
+31(0)634139260
+31(0)647268603
www.prorail.nl
+31(0)634139260
www.timkoomen.nl
� Between 40-60% of IT-projects use scrum
� Always a project manager
� Product owners: usually 2, one big project 5
� Product owner: mostly an information analyst or architect
Number of suppliers: 1-2� Number of suppliers: 1-2
� Number of scrum teams: 1-4
� Test role(s) in scrum team? Usually, but not always. If yes, then usually from supplier, sometimes from ProRail
� Frequency of acceptance testing: from 1 x at the end to almost each sprint
27
Perspective
Status October 2013
Criterium Info
plu
s
PR
L
RR
CB
PV
S
Ge
luid
sre
gis
ter
VO
S
Ve
rvo
ed
ers
po
rta
al
Qu
ova
dis
II
Fri
ts
Fri
ts S
LA
Sp
oo
rweb
1.1 Product owner appointed = Yes
1.2 Stakeholders appointed = No
1.3 Scrum master appointed = Deels
1.4 Team completed (designer, developer, support, test) blanc = Unknown1 Organisation
28
1.4 Team completed (designer, developer, support, test) blanc = Unknown
1.5 Team in 1 room
1.6 Team members act according to expectations
2.1Backlog in use
2.2 User stories described
2.3 Chang control on backlog and US
2.4 Regular meetings stakeholders-PO
2.5Clearly defined sprints
2.6 Each sprint a Sprint Planning Meeting
2.7 Daily Scrum standup meeting
2.8 Scrumboard en Burndownchart up-to-date
2.9 Each sprint a Sprint Review Meeting
2.10 Each sprint a Sprint Retrospective meeting
2.11 After each sprint a shippable product
2.12 “No changes” in sprint stricty maintained
2 Process
