STARTING A DEVOPS JOURNEY IN A TELCO IN BRAZIL
Julio PereiraAdvisory ConsultantDell EMC [email protected]
Ricardo HerrmannData ScientistDell [email protected]
Felipe MiziaraSenior Project ManagerDell [email protected]
Knowledge Sharing Article © 2018 Dell Inc. or its subsidiaries.
Gerson GracianiSenior Solutions ArchitectDell [email protected]
2018 Dell EMC Proven Professional Knowledge Sharing 2
Table of Contents
1 Problem Description .............................................................................................. 4
1.1 Development Tracks .............................................................................................. 5
1.2 Pre-production and Production Tracks ................................................................... 6
2 Our Approach ......................................................................................................... 7
2.1 Future State Workshops......................................................................................... 7
2.2 Interviews ............................................................................................................... 9
2.3 Value Stream Map ................................................................................................. 9
2.4 Center of Excellence .............................................................................................. 9
2.5 Squads ................................................................................................................. 10
2.6 Velocity and Quality ............................................................................................. 11
3 Continuous Delivery Pipeline System ................................................................ 11
3.1 Conceptual Pipeline ............................................................................................. 11
3.2 Architecture and Tools ......................................................................................... 12
3.3 Pipeline in the Cloud ............................................................................................ 14
3.3.1 Git Repositories................................................................................................. 15
3.3.2 Kubernetes installation with Ansible and kubeadm ............................................ 15
3.3.3 The “Hello, world” Continuous Delivery Pipeline ................................................ 16
3.4 Pipeline on Premises ........................................................................................... 17
3.5 Potential Improvements........................................................................................ 18
4 Development Practices (in Theory) ..................................................................... 19
4.1 Feature Flags ....................................................................................................... 19
4.2 Trunk-Based Development ................................................................................... 20
5 Development Practices (in Practice) ................................................................... 22
5.1 Big Bang Approach .............................................................................................. 22
5.2 Componentize Approach ...................................................................................... 24
5.3 And the Selected Strategy Was ............................................................................ 24
6 Conclusions ......................................................................................................... 25
2018 Dell EMC Proven Professional Knowledge Sharing 3
List of Figures
Figure 1: Tracks ........................................................................................................... 5
Figure 2: Development Tracks ..................................................................................... 6
Figure 3: Pre-production and Production Tracks .......................................................... 7
Figure 4; Shared Vision between Leadership and Delivery Teams ............................... 8
Figure 5: Continuous Delivery Common Characteristics .............................................. 8
Figure 6: Center of Excellence ................................................................................... 10
Figure 7: Conceptual Pipeline .................................................................................... 12
Figure 8: Tool Set in Use ........................................................................................... 13
Figure 9: Proposed Tool Set ...................................................................................... 13
Figure 10: Basic Architecture ..................................................................................... 14
Figure 11: Dashboard for the “Hello, world” Pipeline .................................................. 17
Figure 12: Final Dashboard ........................................................................................ 18
Figure 13: Work Around to Allow Downloading New Images ...................................... 18
Figure 14: Anatomy of a very simple Feature Flag ..................................................... 20
Figure 15: Flow vs. Trunk-Based Development .......................................................... 21
Figure 16:Big Bang Strategy ...................................................................................... 23
List of Tables
Table 1: Comparing complexity .................................................................................. 20
Table 2: Strategies to Achieve Trunk-Based Development ........................................ 21
Disclaimer: The views, processes or methodologies published in this article are those
of the authors. They do not necessarily reflect Dell EMC’s views, processes or
methodologies.
2018 Dell EMC Proven Professional Knowledge Sharing 4
1 Problem Description
Seeking to keep up with competition in highly challenging Brazilian telecommunication
market, our Client established a development process occurring in separate parallel
tracks.
To better understand common issues one must take a close look at the complexity of
delivering a software (e.g. a web portal for over 90 million subscribers) which
integrates several systems (legacy and new). Each of these systems are likely to have
a separate and different life cycle, so it is clear that different parts undergo different
development tracks.
Each track represents a full copy of source code1 (i.e. this is not branching technique),
and due to integration idiosyncrasies, it also represents a separate environment. The
main reasons behind this idea are:
• Development highly outsourced, occurring mainly outside Client’s properties.
• Business Areas pushing many concurrent changes in existing software systems.
• The need to sustain market share.
• Coexistence with legacy systems, many of them mainframe-based.
• Large databases imposing serious issues during system integration phases
(subscriber base is around 100 million).
Apart from the fact that parallel tracks offer some comfort, merging code proved to be
a complicated task: i.e. loss of quality, large number of human resources dedicated to
perform integration tests and rework.
Currently, the Client implemented 5 parallel tracks (Figure 1) to allow matching
corporation release dates:
1 Source code was kept in SVN, however, rather than using SVN as a full capacity source code control, it was merely used
to keep whole copies of each track – at the end of the day, SVN had to be replaced by Git.
2018 Dell EMC Proven Professional Knowledge Sharing 5
Figure 1: Tracks
• Tracks 1 to 3, to accommodate development and acceptance tests of new
features.
• Pre-production, where regression tests and hot fixes occur.
• Production, and eventually emergency fixes (which are replicated manually to
pre-production and downwards development tracks).
1.1 Development Tracks
Each development track is created following a fixed schedule in order to match release
deadlines by cloning production code.
Following this logic, features are planned according to each track. Since each release
takes approximately 60 days, an acceptance tests takes place every 15 days (see
section 2.3).
Note that tracks are completely independent. Therefore, when development
successfully finishes, code must be merged into pre-production track. A track is
destroyed after merging process is completed and recreated further on (Figure 2).
2018 Dell EMC Proven Professional Knowledge Sharing 6
Figure 2: Development Tracks
In this model, it is clear that features are implemented on entirely independent source
codes, and must be planned in such a way that different development teams avoid
touching “common code”. When this situation cannot be avoided, merging tasks grow
with complexity.
1.2 Pre-production and Production Tracks
Pre-production track is used for code merges, user acceptance, regression tests and
hot fixes, and production is eventually used to implement emergency fixes.
This model imposes a catch: each change performed in the pre-production and/or
production tracks must be manually re-implemented in the pre-production track and, of
course, in the development tracks. Since these are most likely implemented while
development is taking place in development tracks, one could say that merging tasks
becomes more complicated as complexity and parallelism grow (see Figure 3).
2018 Dell EMC Proven Professional Knowledge Sharing 7
Figure 3: Pre-production and Production Tracks
2 Our Approach
This section explores actions taken to address the problems described in the previous
section. It is clear that a cultural change is necessary to resolve described issues, so
several steps were taken, some still underway:
• Executing Future State Workshops, where Client’s resources started to be
evangelized on general topics related to DevOps discipline (see section 2.1).
• Performing Interviews, 1-on-1 conversations with selected personnel, particularly
those involved in the pain points this initiative should solve (see section 2.2).
• Sponsoring a Center of Excellence, which is an entity aiming to promote
discussions, education and resolving blocking points (see section 2.4).
• Creating Squads, which are hands-on teams (see section 2.5).
• Implementing a Continuous Delivery Pipeline System, an integrated and
standardized environment (see section 3).
2.1 Future State Workshops
Future State Workshops were meant to evangelize Client’s personnel, establishing a
common ground of concepts, pinpointing what the organization should look like, and
covering aspects of velocity, quality and value of delivered system software (Figure 4).
2018 Dell EMC Proven Professional Knowledge Sharing 8
Figure 4; Shared Vision between Leadership and Delivery Teams
These workshops provided a forum to bring concepts that DevOps should be a culture
with a common view of success, success criteria and measures, an incremental,
iterative approach, collaborative cross functional teams, and, above all, trust.
The workshop was the initial introduction to the concept of Continuous Delivery and its
disruptive philosophy of bringing together Development and Operations Teams in a
continuous flow (Figure 5).
Figure 5: Continuous Delivery Common Characteristics
Additional covered subjects:
• Pipelines of Continuous Integration/Delivery (see section 3).
• Value Stream Map (see section 2.3).
• Velocity and Quality (see section 2.6).
N.B.: 40 people attended 3 workshop sessions.
2018 Dell EMC Proven Professional Knowledge Sharing 9
2.2 Interviews
Interviews were the perfect occasion to establish a relationship of trust with personnel
suffering daily pressures of a complicated development/deploy cycle.
Interviews2 generated key input to produce a Current State Report, which presented
Client with important findings to help drive a successful DevOps transformation.
N.B.: In this Project, we were able to interview around 15 key Client’s professionals.
2.3 Value Stream Map
One of the key objectives of our team was to create the Value Stream Map of the
selected application. To elicit required information, a series of workshops were
scheduled. However, since these workshops involved a large number of people from
different areas, getting all those stakeholders in the same room at the same time was
a huge challenge.
The following Client’s Areas were involved: Development, Architecture, Middleware,
Configuration Management, Project Manager Office, Infrastructure, Procurement and
Customer Digital Channel.
Main findings of the Value Stream Map workshops were:
• Release model is time-boxed.
• As expected, there are many manually-driven activities (i.e. environment
preparation, build and deploy, and merges) which could be automated by
implementing a Continuous Delivery Pipeline System.
• Team morale is low, since weekends and nights are commonly spent moving
software to production.
• A huge amount of time spent correcting software caused side effects on portions
of code which were running without bugs.
• IT Department is constantly blamed for low customer satisfaction and, eventually,
customer base reduction.
2.4 Center of Excellence
The Center of Excellence is a perennial organization with the goal to provide
knowledge, best practices and coaching, facilitating the implementation and support
for the DevOps processes and practices (Figure 6).
2 Results were anonymously recorded, so each one could feel comfortable in exploring pain points.
2018 Dell EMC Proven Professional Knowledge Sharing 10
Figure 6: Center of Excellence
Stakeholders from multiple organization teams should participate in Center of
Excellence meetings forming two separate groups: strategic and tactical:
• Strategic, is a committee formed by mostly managers and/or stakeholders, whose
main responsibilities are to resolve general deadlocks, define guidelines and
processes. Meetings are taking place every 15 days.
• Tactical, is a committee formed by mostly key professionals, whose main
responsibility is to actually drive DevOps transformation as defined by the
Strategic Committee. Meetings are taking place every week.
2.5 Squads
A Squad in this context is a multidisciplinary team assembled to implement Continuous
Delivery Pipeline for a specific software system (see section 3). Squads are managed
by the Center of Excellence (see section 2.4) – the basic responsibility of each squad
is to develop and maintain:
• Tests during project timeline.
• Automation for infrastructure and applications.
• The DevOps framework and related integrations.
N.B.: In this Project, we were able to put in place two squads.
2018 Dell EMC Proven Professional Knowledge Sharing 11
2.6 Velocity and Quality
To acquire velocity, IT organizations must go through fundamental changes. However,
quality is essential and must be pursued as well. Critical points to achieving velocity
and quality are:
• Automating testing processes (see section 3).
• Removing bottlenecks in the process of creating value to the customer from the
beginning of product life-cycle.
• Implementing trunk-based development (see section 4.2).
• Creating a discipline of development using the concept of Feature Flags (see
section 4.1).
3 Continuous Delivery Pipeline System
This section describes steps taken to implement pipelines of continuous
integration/delivery:
• Conceptual Pipeline Describes what conceptual pipeline should be (see section
3.1).
• Architecture and Tools Briefly describes our work in analyzing and proposing a
set of tools and underlying architecture (see section 3.2).
• Pipeline in the Cloud One of our first challenges was that our Client did not
provide necessary infrastructure upfront, so we decided to build a conceptual
pipeline off-premises, using a commercial cloud service to serve as Proof of
Concept (see section 3.3).
• Pipelines on Premises Immediately after infrastructure was made available we
migrated the Proof of Concept to our Client’s infrastructure (see section 3.4).
• Potential Improvements General discussion on opportunities to improve and
address common issues (see section 3.5).
3.1 Conceptual Pipeline
A Conceptual Pipeline is, as made clear by its name, a pipeline of continuous
integration/delivery, with steps and quality gates to ensure quality software is deployed
(Figure 7).
2018 Dell EMC Proven Professional Knowledge Sharing 12
Figure 7: Conceptual Pipeline
The conceptual pipeline shown in Figure 7 served as model and, at the same time, as
a road map proposition highlighting important aspects such as:
• Pipelines should support various “states” (pre-commit, commit, QA, acceptance
and production).
• “Everything” is code (application, infrastructure and compliance).
• Quality Gates should also be in place to guarantee if pipeline is healthy, enforcing
only good product is moving forward.
• Compliance is clearly met (for example, COBIT and PCI validation stages).
3.2 Architecture and Tools
An important deliverable was presented as a result of extensive study on which tools
should be put in place to best suit our Client’s needs. We started this analysis based
on a set already in place (Figure 8), and in our Client’s best interest, we proposed a
complementary set of tools that would enable implementation of a Continuous Delivery
Pipeline System (Figure 9).
2018 Dell EMC Proven Professional Knowledge Sharing 13
Figure 8: Tool Set in Use
Figure 9: Proposed Tool Set
Finally, the basic implemented architecture depicted in Figure 10 makes it easy to see
which tools were to be integrated and actually used.
2018 Dell EMC Proven Professional Knowledge Sharing 14
Figure 10: Basic Architecture
3.3 Pipeline in the Cloud
Our project expected the customer to provide the necessary infrastructure to install a
Continuous Delivery Pipeline System. However, since it was not available upfront, we
decided to create an environment in the cloud fully implemented with an “as code”
approach with the sole purpose as Proof of Concept.
This approach proved its value further on, since we successfully moved our
Continuous Delivery Pipeline System to our Client’s premises with “minimal” effort (see
section 3.4).
The following strategy was devised:
• Every tool should be a Docker container, enabling us to avoid the required steps of
its installation and focus solely on the configuration.
• A Git repository should be used for each tool to keep versions of the dockerfile, as
well every configuration file used to create the tool container.
• kubeadm tool to automatically create the Kubernetes cluster.
• Vagrant to start up the whole solution.
• Ansible playbook.
• A cloud provider subscription to hold the servers.
• A “Hello, world” application in Java language managed by Maven.
2018 Dell EMC Proven Professional Knowledge Sharing 15
• Last but not least, a Continuous Delivery Pipeline was created for a “Hello, world”
example application.
3.3.1 Git Repositories
Every container specification, configuration and installation commands were saved in a
Git repository, allowing us to keep track of every change made to the environment:
• framework-vagrant.git. Contains the Vagrant configuration to create the four nodes
used in the solution all together with the provisioning script that installs the whole
solution. This provisioning script does the following tasks: clones Docker
repositories, builds every necessary Docker image, installs the Kubernetes cluster,
and performs the startup of the tools (e.g. Jenkins, Nexus, Grafana, InfluxDB, etc.).
• framework-kubernetes.git. Contains the Ansible playbook which installs the
Kubernetes cluster.
• framework.git. Contains the Kubernetes configuration files that starts the used
services.
• framework-jenkins.git. Contains the dockerfile, configuration files, and test scripts to
build, configure, and test the Jenkins container. Each configuration is specified in
the Groovy Language. No manual configuration is made after the Jenkins server is
running.
• framework-influxdb.git. Contains the dockerfile, configuration files, and test scripts
to build, configure, and test the InfluxDB database container. The outcome is an
InfluxDB with a database called pipeline.
• framework-grafana.git. Contains the dockerfile, configuration files, and test scripts
to build, configure, and test the Grafana dashboard container. The outcome is a
Grafana server with pre-configured dashboards.
• framework-gitlab.git. Contains the dockerfile, configuration files, and test scripts to
build, configure, and test the GitLab container.
• framework-nexus.git. Contains the dockerfile, configuration files, and test scripts to
build, configure, and test the Nexus container.
• framework-sonarqube.git. Contains the dockerfile, configuration files, and test
scripts to build, configure, and test the SonarQube container.
3.3.2 Kubernetes installation with Ansible and kubeadm
An Ansible playbook was implemented to:
• Install all operational system prerequisites.
2018 Dell EMC Proven Professional Knowledge Sharing 16
• Install Kubernetes through the kubeadm utiliy.
• Build every container tool container (Jenkins, SonarQube, Nexus, etc).
• Deploy the Jenkins container in the Kubernetes cluster.
3.3.3 The “Hello, world” Continuous Delivery Pipeline
A “Hello, world” Java application was created to demonstrate how a Continuous
Delivery Pipeline should function.
The pipeline (Figure 11) is composed of the following Jenkins jobs:
• Commit. This job runs after a developer gives a git push command. The git push
command sends all code modification made by the developer to the central
repository. A webhook is triggered automatically, starting the job. The following
steps happen inside the commit stage: build, unit test, static code analysis,
packaging, and publishing to the Nexus artifact server. As a final step, the commit
stage calls a job to publish the new application version to the Infra Job.
• Infra Job. This job runs after any modification to the Infra Git repository occurs or
by being triggered as the last step of the commit stage. It is responsible for
creating a Docker container that contains a Java Tomcat Web Server and installing
the pre-built application inside it. After the Docker image is ready, it is then
published to the Nexus registry server. As a final step, the infra stage calls a job to
publish the new container version to the Dev Deploy Job.
• Dev Deploy Job. This jobs runs after being triggered by the infra repository. It
deploys the new created docker container as a Kubernetes deploy and a
Kubernetes service to access the web interface of the container. Several tests are
then executed to ensure the image and the software are functioning properly.
After the tests are successful, the E1, E2, E3 and PP jobs are called in parallel3.
• E1, E2, E3 and PP Deploy Jobs. These are similar to the Dev Deploy Job – actually
performing a deploy in a specific environment as well performing several tests to
ensure the application is working properly. E1, E2 and E3 Jobs finalize with no
additional steps, and PP Job starts a Prod Deploy Job as its last step.
• Prop Deploy Job. This is the actual final deploy in production, which creates a
Kubernetes service to access the newly deployed application.
3 En stands for tracks 1 to 3 (see section 1).
2018 Dell EMC Proven Professional Knowledge Sharing 17
Figure 11: Dashboard for the “Hello, world” Pipeline
3.4 Pipeline on Premises
Just after the on-premises infrastructure was made available, a Continuous Delivery
Pipeline System was installed. However, a new challenge emerged: per our Client’s
policies, no Internet access was allowed, impeding the installation of Docker containers.
With the project time line under severe pressure, a workaround was devised.
Containers were downloaded and created by connecting a laptop outside Client’s
properties, and finally brought each container as a Docker registry server manually
pushed into the local Nexus repository.
A script was then created to install Kubernetes using these containers from the Nexus
repository. Doing so enabled us to use all the tooling Git repositories to build and
install the tools inside the Kubernetes cluster.
A slight improvement was made to the dashboard – now users may select which
Software Factory (“Fábrica”), Application (“Aplicação”) or Component (“Componente”)
they are willing to watch (Figure 12).
2018 Dell EMC Proven Professional Knowledge Sharing 18
Figure 12: Final Dashboard
3.5 Potential Improvements
While our strategy prescribes full Internet access in order to download images to be
used by the Continuous Delivery Pipeline System, we understand many corporations
may consider this a potential safety breach.
Initially, one may look at this problem with the biased opinion that Information Security
Department “likes to play in the safe zone”. However, the true reason is that all
departments should be involved from the very beginning, creating a culture where risk
is thoroughly discussed, and necessary actions are agreed upon.
In our experience, this was a tough issue that actually put the project plan in jeopardy,
so we believe our initial approach should have proposed a compromise such that
necessary components were made available by a “special task force” with sufficient
privileges to access the Internet to build whatever is necessary (Figure 13).
Figure 13: Workaround to Allow Downloading New Images
2018 Dell EMC Proven Professional Knowledge Sharing 19
Though obvious now, proposing such an alternative during our first meetings with the
Information Security Department might have saved a lot of time and effort.
4 Development Practices (in Theory)
In this section, we discuss two software development best practices that are enablers
of enterprise grade continuous deployment models, and which help the development
team to directly address two of their main pain points.
One of the consequences of the old development model is that developers (from
multiple companies) periodically had to work during weekends, late shifts and, in some
cases, pull all-nighters, in order to push features into production or merge changes into
testing environments at the right time. Both feature pushing and code merging can
benefit from a simple software development practice, namely the introduction of
Feature Flags (or Toggles [17]) in the applications’ code.
There may be a spectrum of Feature Flags characteristics [17], but two distinguishing
dimensions are their activation dynamism and their span of life (“longevity”). Two main
groups in this space immediately serve our purposes:
1. Short-lived, statically-configured Feature Flags are enablers of the so-called
Trunk-Based Development practice, which avoids excessive repository branching
and merging; and
2. Long-lived, strategy-activated flags enable behavioral changes in a running
application, turning features on/off at the right time, for the right users, with no
downtime and hands-free operation.
We will discuss the theory behind both practices in the sub-sections below and how we
applied them in practice in the subsequent section (Sec. 5).
4.1 Feature Flags
Developing software using Feature Flags allows developers to conditionally execute
alternative code blocks – whether encapsulated within if-then-else statements (Figure
14) or using polymorphic dispatching –to better control software releasing. The
practice of using feature flags while deploying software with new features that are
initially turned off but that will be turned on at a future point (manually or automatically)
is also called dark launching.
2018 Dell EMC Proven Professional Knowledge Sharing 20
Figure 14: Anatomy of a very simple Feature Flag
This technique – a central component of the DevOps practice – allows for granular
control of features going into production. Its main characteristics are:
• Enables developers/operators to modify system behavior without changing code
• To deliver new functionality to users rapidly but safely, as features can be turned
off quickly in face of problems.
There are also a few downsides in their use since feature flags introduce complexity
and technical debt, which easily accumulate if not dealt with in time. When developing
in this manner, developers should remember to:
• Constrain the number of toggles
• Guarantee that flags should be queried only once
• Remove flags as soon as possible
To clarify these concepts, Table 1 presents a simple comparison between concerns
developers should be aware of when working using Branching Codebases and Trunk-
Based Development.
Table 1: Comparing complexity
Technique When developing... When releasing...
Branching Codebases Changes being made. Merging, testing, and
rewriting a lot of code.
Trunk-Based
Development
Changes being made and
how these changes can be
safely deployed.
No work.
N.B.: As Proof of Concept we implemented a feature flag example in the “Hello,
world” Java application (see section 3.3.3) using Togglz.
2018 Dell EMC Proven Professional Knowledge Sharing 21
4.2 Trunk-Based Development
When multiple teams are changing the same software module, the first one that
reaches final user approval must be promoted to production. As this process is being
repeated for each new version of the same module, and since testing is not
systematically performed, it is easy to perceive that this new version of the same
module will probably disrupt the version that was in production. This is commonly
called “merging hell”.
One technique often used to avoid merging hell, is Trunk-Based Development, where
software branching is eradicated from the development process. Therefore, each
developer actually works in the main trunk of the source code. Each change must be
tested and checked so modules are always updated and merging conflicts are early
identified and corrected (Figure 15).
Figure 15: Flow vs. Trunk-Based Development
The concept seems simple, but its implementation might be tricky since it had to be
applied to a live software system. In other words, we had to provide options that would
allow moving from the development in tracks adopted by our Client (see section 1).
With this in mind, two strategies were presented (Table 2):
Table 2: Strategies to Achieve Trunk-Based Development
# Description Pros Cons
1 Big Bang, i.e. convert
each track to Trunk-
Based Development in
waves according to its
release date (see
section 5.1).
Nothing is left behind
and could be
implemented in quite
short time span.
Developers not trained
in Feature Flags and
Trunk-Based
Development
techniques could spoil
the whole thing.
2018 Dell EMC Proven Professional Knowledge Sharing 22
2 Convert by
modularizing the code
(see section 5.2).
Smaller portions of
code would be moved
into Trunk-Based
Development, low
impact in current
working flow.
Software architecture
had to be redesigned
(since it was not
modularized).
“Trunk-based development is likely the most controversial practice
discussed in this book [...] However, the data from Puppet Labs’ 2015 State
of DevOps Report is clear: trunk-based development predicts higher
throughput and better stability, and even higher job satisfaction and lower
rates of burnout.” [2]
5 Development Practices (in Practice)
While planning development strategy migration paths alongside the Client, we were
faced with a few possible scenarios, each of which exhibiting different tradeoffs. We
discuss the main aspects of our decision below.
5.1 Big Bang Approach
This approach proposes the following (Figure 16):
1. Migrate pre-production track from SVN to Git (historical information will then be
kept in SVN, only to be used as reference).
2. For each track, at the end of its each deployment and acceptance cycle, also
migrate to Git (i.e. checking-in code to Git instead of SVN). Traditional branching
technique still applies.
3. While Git conversion is taking place, implement automation for build, unit testing,
compliance and change management.
4. After all three development tracks are finished converting to Git – and only then –
start Trunk-Based Development (this also means that Feature Flags must have
been absorbed by the development team).
2018 Dell EMC Proven Professional Knowledge Sharing 23
Figure 16: Big Bang Strategy
N.B.: Production track, after conversion, will contain only deployed code. Therefore,
no “source code” would be kept for it.
General considerations:
• Conversion occurs in a relatively short period.
• Only one pipeline shall be created.
• Drives the whole team to practice quality gates for any new feature, or
maintenance for existing ones.
• Manual build and deploy tasks are kept to a minimum.
• Measurement of unit test, code static analysis and compliance made possible for
the whole system.
• Merging hell is eliminated very early in the process.
• Ceases the growth of technical debt.
• There will be higher effort and complexity.
• It will be necessary to massively resolve eventual issues, probably indicating that
a larger team should be put in place.
• All development teams will be affected, however expected transition time will to
be short.
• Probable productivity reduction in the beginning due to necessary change in the
development culture, especially as a side effect of the implementation of quality
gates.
• The whole team will face the complexity of applying Trunk-Based Development
and Feature Flags.
2018 Dell EMC Proven Professional Knowledge Sharing 24
5.2 Componentize Approach
As an alternative, this approach forebodes a transition in steps by “extracting” parts of
the whole system into smaller components. Each new component is then migrated to
Trunk-Based Development using Feature Flags technique as needed.
In this case, software system architecture must undergo an important change by
creating libraries of common code which will be used by “more specialized”
subsystems. This construction will be in the following phases:
1. Componetization.
2. Implementation of continuous integration for the common code.
3. Implementation of continuous delivery for generated artifacts.
General considerations:
• Allows for a fast delivery upfront, since it will be applied for an isolated part of the
application.
• Requires a long term planning in order to allow partial migrations of source code
to the new model.
• Imposes a different method of work for each portion of the source code.
• Isolation of the source code will lower the risk and will serve as “lessons learned”
for the future work.
• Will not create a DevOps culture upfront.
• Each development track will likely have its own independent code.
• It will be necessary to implement five different pipelines (production, pre-
production, tracks 1 to 3).
• Merging code complexity will be likely increased.
• There will be a high effort in changing the entire software architecture.
5.3 And the Selected Strategy Was...
Considering the software system complexity and the fact that development is currently
being performed by a third party, our Client chose option #2. We believe the main
reasons for this choice were:
• Fear of applying such change in culture.
• Unfamiliarity with the advocated techniques (Trunk-Based Development and
Feature Flags).
2018 Dell EMC Proven Professional Knowledge Sharing 25
• Criticality of the software system under analysis.
• Poor software architecture.
• Pressure from the third party in charge of the development.
6 Conclusions
It is clear that velocity and quality are key factors in gaining and sustaining customer
base, especially in a challenging market such as telecommunications, where
competition is fierce. Today, customers anticipate and expect the same experience as
they receive from Amazon, Uber, Airbnb and so on – and they expect that from the
same enterprises that provide access to these services.
These enterprises are pursuing velocity and quality by moving traditional software
development life cycles to another level, breaking barriers from teams that used to
work apart – from conception to operation, from ethereal definitions to actual product
delivering, changing the concept of “done” to something that provides the final user
with quality.
Citing just a few known issues is easy and most certainly common to usual product
releasing: no source code control, no systematic testing, no traceability of changes,
conflicts generated by multiple parallel software developments and poor system
architecture.
Particularly when software development occurs on uncontrolled multiple parallel
environments, situations derail what is usually called merging hell, when changes must
be finally converged to what is really going into production. This can be avoid by using
Trunk-Based Development.
As developers converge to Trunk-Based Development, issues may appear that are
related to deployments of evolving features in the same modules upcoming integration
situations. This can be avoided by using Feature Flags.
Of course, it is not easy to shift from development being performed on branches to
Trunk-Based Development and Feature Flags, as this requires a transformation in
culture and establishing a mind-set of writing code with quality.
The final step to achieve quality and velocity is to implement a Continuous Delivery
Pipeline System, where each new software feature is created, logged in the source
code tool, compiled, tested and automatically deployed in an ephemeral, immutable
and idempotent infrastructure.
It is necessary to point out that engaging personnel from our Client was one of the
toughest jobs, so we advise that an effective communication plan be put in place and
several meetings and workshops must occur to evangelize all hierarchy levels.Keep in
mind that we are advising a deep change in development methods and techniques,
2018 Dell EMC Proven Professional Knowledge Sharing 26
introducing fundamental remodeling in software architecture, and advocating that
barriers should never exist in a corporation.
Last but not least, such profound transformations are likely to directly impact how third-
party software factories are contracted, measured and/or remunerated.
References
[1] Vincent Driessen. A successful Git branching model
http://nvie.com/posts/a-successful-git-branching-model/
[2] Gene Kim, Jez Humble, Patrick Debois and John
Willis. DevOps Handbook
[3] Ansible. Simple IT Automation
https://www.ansible.com/
[4] Docker. Software technology providing containers
https://www.docker.com
[5] Java. The Java Programming Language
https://java.com/
[6] Jenkins. Continuous Integration and Continuous
Delivery https://jenkins.io/
[7] Git. Git is a free and open source distributed version
control system https://git-scm.com/
[8] GitLab. Web-based Git repository manager
https://gitlab.com/
[9] Grafana The open platform for analytics and
monitoring https://grafana.com/
[10] Groovy. The Apache Groovy programming language
http://groovy-lang.org/
[11] InfluxDB. Time Series Database Monitoring &
Analytics https://www.influxdata.com/
[12] Kubernetes. Open-source system for automating
deployment, scaling, and management of
containerized applications https://kubernetes.io/
[13] Maven. Apache Maven is a software project
management and comprehension tool
https://maven.apache.org/
[14] Nexus. Open Source Governance for DevSecOps
Innovation https://www.sonatype.com/
2018 Dell EMC Proven Professional Knowledge Sharing 27
[15] SonarQube. Continuous Code Quality
https://www.sonarqube.org/
[16] SVN. Apache Subversion
https://subversion.apache.org/
[17] Pete Hodgson. Feature Toggles
(aka Feature Flags). October 9,
2017. https://martinfowler.com/articles/feature-toggles.html
[18] Togglz. Feature Flags for the Java platform
https://www.togglz.org/
[19] Vagrant. Tool for building and managing virtual
machine environments in a single workflow
https://www.vagrantup.com/
2018 Dell EMC Proven Professional Knowledge Sharing 28
Dell EMC believes the information in this publication is accurate as of its publication
date. The information is subject to change without notice.
THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” DELL EMC
MAKES NO RESPRESENTATIONS OR WARRANTIES OF ANY KIND WITH
RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY
DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A
PARTICULAR PURPOSE.
Use, copying and distribution of any Dell EMC software described in this publication
requires an applicable software license.
Dell, EMC and other trademarks are trademarks of Dell Inc. or its subsidiaries
