Interview: Calvin Powers, with Gene King, Ann Marie Fred and Michael Elder
POWERS: Hi, I'm Calvin Powers, I'm the managing editor
for security at IBM developerWorks. DevOps has been a big
movement in the IT industry, and it's getting a lot of
coverage on developerWorks especially in our Agile
Development Zone. And we've got a very active Enterprise
DevOps Blog which you really ought to check out if you get a
chance.
But when I attended AppSec USA a couple of months ago --
back in the fall -- I had the pleasure of meeting and
hearing Gene Kim speak at that event, and he impressed me
with his ability to connect IT security with the DevOps
movement. And so, I asked him to expand on that a little
bit and he has just published a series of articles on
developerWorks about his views on DevOps.
As you know, Gene Kim was the founder and CTO for Tripwire
for 13 years, and it's that 13 years of experience that led
him to writing his most recent book, which is called The
Phoenix Project -- let me hold it up here.
[LAUGHTER]
The Phoenix Project. And you should run out and buy it.
And so I thought we would bring Gene Kim on to have a bit of
a discussion with us. And to help me out and help put
together a bit of a roundtable, I asked two of IBM's DevOps
subject matter experts to join us on the call today.
-1-
Interview: Calvin Powers, with Gene King, Ann Marie Fred and Michael Elder
First, we have Ann Marie Fred, she's a software developer in
Tivoli and is an active contributor to our Enterprise DevOps
Blog. Ann Marie, welcome to the roundtable and tell us a
little bit about yourself.
FRED: Hi, nice to meet everybody. I'm Ann Marie
Fred, and yes, I've been working for IBM for about 15 years
now and I've been in the cloud area for the last three
years. Most recently, I've been working on two projects:
one was SmartCloud Continuous Delivery, which is our
continuous delivery/DevOps based offering; and also, I'm
working on some code for SmartCloud provisioning.
POWERS: All right. And we also have with us today, we
have Michael Elder. He is a senior technical staff member,
he's in our Rational brand. And he leads the product
offering for IBM SmartCloud Continuous Delivery, which helps
our enterprise customers begin adopting DevOps practices.
Michael, welcome to the roundtable, and tell us a little bit
about yourself.
ELDER: Sure. Hi, everyone. Again, my name's Michael.
I'm responsible for the SmartCloud offering from Rational
and Tivoli, which is a joint project. And specifically, our
goal was to enable various enterprise customers to adopt
small practices for DevOps that we thought they could
-2-
Interview: Calvin Powers, with Gene King, Ann Marie Fred and Michael Elder
contain.
You know, the book talks a lot about some of the cultural
challenges and the adoption hindrances that kind of get in
the way, so our goal is to come up with a couple of things
that we thought would be sort of simple, achievable
incremental steps that we could do.
So, I work with Ann Marie as well as several other
developers both in Rational and Tivoli in this space, and we
integrate with the various cloud platforms from IBM to
support continuous delivery directly from the developer
process.
POWERS: All right. So let's bring Gene in. Gene,
welcome to the roundtable. Tell us a little bit about what
motivated you to write this book. I'm going to hold it up
again, The Phoenix Project, a novel about IT DevOps and
helping your business win. Give us a little bit of history
about this book, Gene.
KIM: You know, I started studying a group of
organizations that back in 1999 we used to call geniuses of
people with great kung fu. [For the other] people that
acted differently, talked differently, but most importantly
they had profoundly different operational results than your
typical organization.
-3-
Interview: Calvin Powers, with Gene King, Ann Marie Fred and Michael Elder
And so over the years, working with the Institute of
Internal Auditors and working with the Software Engineering
Institute, they helped me kind of coin a better name for
these groups. And we call them now the high-performing IT
organizations that have the best feature flow rates, the
best operational stability, the best security and the best
posture compliance.
And you know, what we found is that these organizations
think so differently, act so differently and get results so
profoundly better than your typical IT organization. And
this book was really meant to sort of show how we understand
what non high-performing IT organizations look like and it's
almost all the same.
But when you have cultural warfare going on between
development, IT operations and information security, you're
almost doomed to failure -- that bad outcomes are almost
preordained.
POWERS: Fair enough.
KIM: And in the first part of the book, yes, it's
like some of the feedback we've gotten through the book is
like, wow, you've just described either the organization I'm
in or the organization I was in prior. And I think that was
because we carefully constructed that novel to show that
-4-
Interview: Calvin Powers, with Gene King, Ann Marie Fred and Michael Elder
yes, these things are exactly the outcomes that happen when
you don't have dev, ops and security working together.
POWERS: Fair enough.
KIM: Yes, that was the first half of the book. And
the other half is really meant to show kind of what are
those breakthroughs that you can get once you get those
different functional areas working together. And what are
the patterns you can put into place that actually are not
mammoth projects but are small things you can do differently
that create far better outcomes.
And so much of that is now being folded into the DevOps, I
would say "body of knowledge" and now we're trying to show
not only what the value of doing that is but how you do
that.
POWERS: Now, you used a word there that doesn't usually
get used when we're talking about IT security. You used the
word "novel," and it's really interesting to me that the
book is...it's an educational book but it is framed and
portrayed and written as a novel with a narrative and plot
and characters and everything rather than a textbook about
DevOps. Why did you and the other authors choose to do
that?
KIM: Well, I would say there were two reasons. One
-5-
Interview: Calvin Powers, with Gene King, Ann Marie Fred and Michael Elder
is that the book was very closely modeled after a book that
we studied for over 10 years, which is called The Goal by
Dr. Eliyahu Goldratt. It was written in the 1980s, and it
was about...it was a novel about a plant manager who had to
fix his cost and due date issues in 90 days otherwise
they're going to shut the plant down.
You know, this book probably most profoundly affected my
professional career, and even though I had never worked in a
plant, certainly never managed a plant, for me it's just...I
couldn't help but walk away thinking there were some
important lessons here that we wanted to take away.
The second thing that I think sort of validated our desire
to do this is that we found that storytelling is the most
effective mode of communication -- that storytelling
actually bypassing all the rational parts of the brain. And
if you want to get those neurons firing, you know, it's
actually through stories that you know, the human brain is
almost sort of designed to sort of understand and be
receptive to.
And so, our goal was to show that we understand the problem
and there is a better way and to take the reader along a
journey to make sure that they feel like these problems that
we're describing are relevant to them. And I'd love to ask
Ann Marie and Michael, you know, hopefully the book was able
-6-
Interview: Calvin Powers, with Gene King, Ann Marie Fred and Michael Elder
to sort of trigger some of those sort of sympathetic
reactions?
FRED: Yes, I liked the novel format. I found that it
was a fast read, and I think it's because I was really
engaged in it and just kind of letting it flow. Yes, and to
me it just sort of pointed out a lot of areas where I want
to learn more myself.
ELDER: I would definitely echo those comments. I
think that when you read through it and you see the
frustrating parts, you can really relate to it and that
makes it more real, right? It's not just a storyline, a
fictional line; it's something that you can kind of say, I
knew that guy in a prior role, or I know that person now and
how they impact the project in either positive or negative
ways. So that was...it was kind of neat to sort of see the
life of what we do made into a format that was entertaining
and dramatic as opposed to technical and boring, so.
KIM: In fact, I mean, as we were writing it, I mean,
I'm sure there were times we were writing a person that we
were [LAUGHTER] like that person named Brent, right? The
person who, no outage can be fixed without Brent, no major
project can be done without Brent, because Brent is always
in the way. Right? I mean, I think most of us have...will
be familiar with that person and sometimes have even been
-7-
Interview: Calvin Powers, with Gene King, Ann Marie Fred and Michael Elder
that person.
POWERS: ...in particular the fact that there were these
moments where you fixed the problem and then he had no idea
how he actually did it.
KIM: I know those people that, they pull it out, and
then after that moment of clarity it vaporizes back into the
nothingness.
POWERS: Right.
KIM: I love that phrase, like every time that
happens Brent gets a little smarter and the organization
gets a little dumber.
FRED: Yes, actually we had an animated discussion
about that at lunch a couple of days ago about what's the
best way to deal with the Brents in your organization and
you know, one person was arguing, well, you can't slow them
down, you have to let them keep doing what they're doing.
And other people were saying, no, you have to make sure
everything that they do is documented. So that really spoke
to me as well.
KIM: And by the way, the irony is you have to do
both, right? I mean, in order for us to get the most out of
-8-
Interview: Calvin Powers, with Gene King, Ann Marie Fred and Michael Elder
Brent and keep Brent focused on the highest leverage, work
that only Brent can do, we have to make sure that Brent is
not doing things that he shouldn't be doing, right? Like
punching the hole in the boat and then having to fix it the
next morning.
So I just think it's kind of one of those grand ironies that
in order for Brent...and we say this with love and
compassion, right? I mean, because some of this is even,
all of us have probably been Brent, is that in order for the
organization to succeed and Brent to be happy, we need to
surround him with the right processes and standardized work
instructions so that Brent doesn't waste his time with
things that Brent shouldn't be doing.
POWERS: Let's get into the...let's get into the common
[cool] mysticism part of the book. Gene, tell us about the
three ways of DevOps.
KIM: Well, so one of our goals was...actually, one
of the complaints that I think is actually valid is that
DevOps [INAUDIBLE] actually say what DevOps is. And so, one
of the things that we want to do is actually codify the
principles that you could derive all the DevOps patterns
from.
And so, in the book there's this character, this Yoda-like
-9-
Interview: Calvin Powers, with Gene King, Ann Marie Fred and Michael Elder
character, this Mr. Mioggi like character named Eric who
helps coach the protagonist in the book. And you know, so
he speaks in these kind of platitudes but I think they're
very important principles that really show kind of what the
underlaying motive for DevOps is.
The first way is all about understanding the flow of work as
you go from left to right, from development to IT
operations. And the question is, why dev and IT ops? And
it's because that's what's between the business and the
customer. Right? And so making sure that we see the flow
of work, the work should only go in one direction: forwards,
never backwards.
The second way is the reciprocal: how do you get the right
feedback loops created from IT operations into development
so that the goal of any process improvement methodology and
philosophy is to shorten and amplify feedback loops. So how
do we take the key learnings that we learned kind of at the
sharp end of the spear in IT operations and get those
embedded into development so that we can prevent those
things from happening again? Or if that can't happen,
certainly we can detect and correct for it next time it
happens.
And the third way is all about creating a culture of
continual experimentation, continual risk taking and
-10-
Interview: Calvin Powers, with Gene King, Ann Marie Fred and Michael Elder
understanding that repetition is a prerequisite to mastery.
And one of the things that I wish I had read and learned 10
years ago is that whether we're talking about Splitz
training or special forces in the military, or learning a
musical instrument, you know, repetition matters.
And so, it's better to practice a musical instrument 15
minutes a day than it to practice three hours once a week.
And so, too, there's certain things that as information
security professionals, as IT operations, we need to be
practicing all the time, whether it's deployments, recovery
work, disaster recovery, you know, penetration testing, you
know, these can't be done just once a year because that
doesn't actually materially change how we do work. So,
repetition creates habits; habits create changed outcomes.
And so, it's from those kind of a very obscure
mystical-sounding principles that Bill, the protagonist, is
actually able to form his own breakthroughs and you know,
create the DevOps patterns in his own organization without
anyone actually having to tell him exactly what to do. It
wasn't too obtuse, you know, for the readers.
POWERS: Well, let's ask. Michael, do you see any
evidence of those three ways in the approach you guys take
on the SmartCloud offerings?
-11-
Interview: Calvin Powers, with Gene King, Ann Marie Fred and Michael Elder
ELDER: Definitely. The biggest one for us is focusing
on the first and second items, right, providing a way that
you can do a continuous deployment from the development line
all the way into at least a testing QA environment. And
then trying to amplify the feedback and the quality
verification, make sure that what was delivered was not
causing regressions, was actually meeting the future
specifications.
You know, the story line in the book where many rapid fixes
delivered on top of each other back to back in very
compressed, crunched, unrealistic timelines actually creates
more chaos instead of reducing chaos.
And that's I think the biggest aspect of trying to amplify
the feedback loop in a way that's controlled, so that when
you do get a positive result, the next thing you do is a bit
more positive, a bit more stable, a bit more improved. And
I think that, to me, is the biggest item. It's exactly the
kind of thing we focus on in the product deliverables that
we produce as well.
FRED: Yes, and I would say, you know, we went through
this a few years ago with adopting agile practices, and now
we're kind of in the middle of pushing the DevOps principles
through our organization as well. I would say it's kind of
a daily battle -- like it's very easy for people to slide
-12-
Interview: Calvin Powers, with Gene King, Ann Marie Fred and Michael Elder
back into the old way of doing things, so...
POWERS: Right.
FRED: ...you really have to be disciplined and
constantly pushing, like this is the right way to do things,
we're doing this for a reason. You know, and these are the
things that are actually going to save you time.
And it's also, it's kind of a trial and error process. Like
some things you think will save time end up just being extra
paperwork; some paperwork you drop because you thought you
didn't need it but then you find out that everything falls
apart.
So I think it's kind of a constant learning experience. But
you know, we also have to learn from each other. There are
other people who have done this before, so we're not
starting from zero.
POWERS: Fair enough.
FRED: It's really a discipline, yes.
KIM: One of the things I just love about continuous
integration and continuous delivery is that...continuous
deployment, is this notion that you have a deployment
pipeline and when things go wrong you stop the deployment
pipeline, right? And the ideal, right, no new features
-13-
Interview: Calvin Powers, with Gene King, Ann Marie Fred and Michael Elder
until we're in a deployable state again.
And in a previous lifetime, I remember being in an
organization, or I had a friend who was in an organization
where, you know, he/she had, you know, a broken build
system, right, where, you know, we...and if you don't have
continual builds, then you can't do continuous testing. If
you can't do continuous testing, then you can't do
continuous integration.
And this creates this downward spiral where if integrations
and merges become painful, you do it less frequently and if
we do it less frequently, that means it takes more time.
It's just this horrible downward spiral.
And I think the whole sort of value system around continuous
deployment and continuous integration is so important for
DevOps. I mean, I would say it is an actual prerequisite to
get these kind of fast feature flow and stability that we
want out of DevOps.
POWERS: You know, that's a great segue into our next
topic, because in your book there is explicit references to
lean manufacturing and the theory of constraints. And I
can't help but notice, you know, part of DevOps is
operations, and so I think it's a natural question to ask,
how much are things like lean manufacturing and theory of
-14-
Interview: Calvin Powers, with Gene King, Ann Marie Fred and Michael Elder
constraints and just basic fundamental operations and
research a foundation for the DevOps movement?
KIM: Oh, I think the DevOps movement has been very
influenced by Deming and Goldratt and the lean folks. And I
think, in my mind, right, what I'm hoping is that the novel
and the book pushes forward one more increment, is the
notion that IT operations has more in common with the plant
floor than most people would expect.
And I think everything from code commit, you know, down to
in production, you know, that is not artisan work, that
actually is operations and there's a lot more recurring work
that happens, and that actually has a lot more in common
with like a bill of materials and a bill of resources and
routings. Right?
And so the whole notion that we can sort of create these
repeatable pipelines into production and ideally automate as
much as possible is something that is completely out of the
lean manufacturing playbook.
And you know, just I think one of the things that we try to
do in the book is try to make those mappings very explicit.
And I think, I'm just delighted beyond words that there's
already this affinity between lean manufacturing and DevOps,
and I'm hoping that this book will push it one more step
-15-
Interview: Calvin Powers, with Gene King, Ann Marie Fred and Michael Elder
forward.
POWERS: It seems like there's probably a lot of people
that might gristle at the notion that they're working on an
assembly line in IT. Not that I'm arguing with you, but I
just...I just wonder if you get much pushback on that point.
KIM: Oh, of course. In fact, Bill, the protagonist
in the novel, right, he says, we use our brains, not our
hands, right?
POWERS: Right, right.
KIM: [LAUGHTER] You know? You know, our work is not
repetitive. And, yet, if you take a look at sort of the
cadence of most operations and most deployments, deployment
work, right, there is actually a lot of recurring work that
happens in every project and every deployment.
ELDER: So, on this particular topic, I do think
there's a lot of correlations between how we develop
software and deliver software and how that process works.
For me, the perspective I've always taken is that you look
at when the assembly line became popular with folks like
Henry Ford, they started manufacturing parts that were more
interchangeable, more standardized, as opposed to these
special artisan pieces that only Bob knows how to create and
only if you have the rest of Bob's machine will actually
-16-
Interview: Calvin Powers, with Gene King, Ann Marie Fred and Michael Elder
[fit] together, that actually having a standard part that
you can kind of replace and plug in.
And I think that notion for both software delivery in terms
of the business features and you know, you make the point
late in the book around driving automation to make the
pipeline repeatable, each of those things -- both the
software layer and the automation layer -- become cogs in
that larger machine.
And if you can standardize around them and provide that
process for how you create them, how you improve them, I
think ultimately that by having a better support within the
machine itself, the machine...the overall machine becomes
more elegant, more stable, because each of its individual
pieces are more elegant and more stable. So, I tend to
agree with the facts that there are a lot of correlations
with software delivery and manufacturing in general.
FRED: Yes, it's interesting. I was just working on a
presentation from...with Professor Ron Dattero from Missouri
State University about the Toyota production system and how
that's related to DevOps.
And when I first saw the topic, I thought he was insane. I
was like, what are you talking about? But then as I was
reading through the presentation, I was like, oh, you know,
-17-
Interview: Calvin Powers, with Gene King, Ann Marie Fred and Michael Elder
they're talking about built-in quality, the importance of
people and teamwork, reducing waste, you know, don't let
your builds pile up behind the test team, delivering things
just in time. You know? It was amazing to me how it really
does sort of tie in to everything that we do.
KIM: And I think one of the things that the lean
folks do better than anyone -- even better than the theory
constraints in the Goldratt school -- is saying that the
highest aspirational goal, right, of a plant is single-piece
flow. Right? So that means no inventory, you have a
continual pipeline of work, you know, almost like an
assembly line, no...that means no worker process, that means
no wait time.
And so I think kind of the whole notion of like when you
look at Amazon doing a thousand deploys a day, doing
deployments on demand, I mean, I think that is an ultimate
embodiment of what a lean practitioner calls single-piece
flow. So I totally agree with you, and I think that's one
of the neatest things I learned in my indoctrination into
the lean world.
POWERS: Okay. Let's move on. And Michael, in some
offline discussions, Michael had raised this topic about
getting buy-in from the management hierarchy. You want to
launch us in on that one, Michael?
-18-
Interview: Calvin Powers, with Gene King, Ann Marie Fred and Michael Elder
ELDER: Sure. So, as we have talked to larger
enterprises, one of the constraints around the story line in
the book is that you have a company that while it has larger
revenue stream they still are primarily focused on one large
project. You have many roles that kind of sit together in
the same space; I'll talk about the geos later. But the
fact that they're able to achieve buy-in because they really
have this sort of top-down failure, right? We had to have
the entire train run into the side of the mountain, crash,
burn, flames, everything, before you got some sense of, we
need to do something differently.
And I thought that it was an accurate model that you get
top-down buy-in and then you have to establish trust among
peers. You know, there's the "off-site meeting," as it was
called in the book, where everyone kind of comes together
and develops some sense of vulnerability and trust among
each other. To me, that is probably the largest cultural
adjustments you have to make.
POWERS: In the South we call those "Come to Jesus
meetings."
ELDER: Very much so.
Ultimately, though, I'm curious, Gene, what your perspective
is, whether you believe that you always have to have that
-19-
Interview: Calvin Powers, with Gene King, Ann Marie Fred and Michael Elder
plane crashing into the side of the mountain effect before
you can have meaningful change in the culture; or, if you've
seen other places where you can make those changes without
crashing the stock price into the ground.
KIM: You know, in fact, you've...Michael, you just
sort of shaped what I would call the moral crusade of why we
wrote the book, because there's a formative moment where
while I was with Tripwire I was working with a gentleman
named Eric Passmore; he's the CTO of AOL.
And you know, I've told this story before in front of Eric,
and you know, for the IT operations people, he was the SVP
of global engineering at the time. You know, he had 1,300
developers working for him. And among the IT operations
team, he was "that Eric." Right?
He was the person that sort of guaranteed that operations
would never get what they need, right?
Until, I remember this, we did this off site, and there was
this moment where he said, oh, my gosh, the reason why we
couldn't ship a certain feature was because IT operations
couldn't upgrade the Linux kernel from 2-4 to 2-6 and get
multithreading support.
And he said, oh, that was as much of a reason for a code
-20-
Interview: Calvin Powers, with Gene King, Ann Marie Fred and Michael Elder
freeze, right, as anything development related. And he
realized, wow, at that...almost like with a snap of a
finger, he became IT operations best friend. I mean, he
became "THAT Eric," right, who demolished roadblocks and
then became one of the staunchest supporters of the things
that IT operations needed.
And one of the things that just blew me away was that, you
know, one of the things that he helped champion was just
changing who was doing the packaging and moved it from
operations to development. And by doing that, we took the
deployment time for the AOL.com homepage from like six hours
to like 45 minutes.
And that had such a huge impact on me, because that wasn't a
huge mega project, it wasn't this huge cultural
transformation; it was just swapping who was doing what,
right, and it had this incredible difference in outcome.
And so that had a huge, that was one of the big aha! moments
for me, was that, you know, wow, it doesn't take the plane
running into the ground, right, into the side of the
mountain. It just takes a real understanding of a shared
goal and shared outcome that's larger than development tests
or operations.
And so one of the profound hopes that we have in the book is
-21-
Interview: Calvin Powers, with Gene King, Ann Marie Fred and Michael Elder
that by showing the patterns, these undesirable effects,
right, that people will recognize themselves and their peers
in the book and say, hey, we don't have to crash into the
mountain in order for us to believe that this is pertinent
to us...
That, you know, we can actually have some healthy
discussions around this and then start, you know, pulling
the...pulling back on the stick and gaining altitude long
before the business is actually jeopardized. Did that make
sense, by the way?
ELDER: It does, it does. I think there's an
interesting point, though, that there is still some failure
point that you have to observe to really realize the value
of that closer collaboration, right? Or the gentleman you
described, it was one feature that might have benefitted the
customers, but it wasn't the stock price crashing per se,
but there was this catalyst effect that said, that's why we
need to cooperate more effectively.
POWERS: He had to have an aha! moment.
ELDER: ...more so than the value that it might derive.
POWERS: Interesting.
KIM: And can I just point out one thing about what's
so interesting to me is that so much of the way we behave is
-22-
Interview: Calvin Powers, with Gene King, Ann Marie Fred and Michael Elder
embedded in sort of the way our organizations are rewarded,
right? Typically development is to be rewarded on fast
feature delivery, right? You know, more features delivered,
the better.
And usually that comes at the expense of quality of features
and non-functional requirements. And that means make
changes as fast as we can, it doesn't matter what the
quality is, while operations is motivated by up-time
availability, and that usually means make no changes ever,
right, over my dead body, right? And so...which can be
quickly arranged, in my experience.
And so, because those measurements are so embodied in kind
of how dev and ops are managed, you know, almost everybody
will feel the effects of that tension, right, of like this
chronic conflict that comes from one organization making as
many changes as they can and the other organization being as
resistant to changes as they can be, right?
So, you know, that's why I say those kind of horror stories
are almost preordained, just because of the way the dev and
ops people are managed.
ELDER: I completely agree with that. I think that we
see that fundamental challenge over and over again, and
personally I tend to think that as long as there's a divide
-23-
Interview: Calvin Powers, with Gene King, Ann Marie Fred and Michael Elder
between the director and VP level of these orgs, you're
always going to have that inner conflict.
I mean, one of the things in the book near the end, maybe
not essentially a spoiler, but there's this notion that you
sort of have to centralize a role around operations that has
impact on both development and operations as that single
unit that's rewarded based not on what they do individually
but how they compete effectively against their competitors
in the marketplace and more so how they please users, right,
as a team.
FRED: And I mean, we found that on our own team we
had to sort of embed some operations people within our
development team in order to really learn from them, you
know, how it is that they get their jobs done. And also, we
had to go put some developers on the operations team. And
boy, most of them go kicking and screaming, actually,
especially when they realize that they have to own pagers.
Nobody really wants to do that.
But it's a very valuable learning experience, and we're
trying to do more of that so they get that experience. And
then they'll go back and they'll say, okay, here's what we
did wrong in development, here are the things we need to
fix. And then they sort of become evangelists for that
transformation, you know?
-24-
Interview: Calvin Powers, with Gene King, Ann Marie Fred and Michael Elder
KIM: Yes, provided they don't go native, right?
[LAUGHTER]
POWERS: Well, let's talk about the line employees a
little bit, the guys that sit in front of the screens in the
data center all day and the developers who write code all
day. How is this going to, how is this DevOps approach
affecting them? I mean, aren't they sort of at ground zero
for all of the chaos?
KIM: Yes, most certainly. In fact, in the book, you
know, the person we created to embody that person is Brent,
right? Brent is...and it's interesting, when I was looking
at the Amazon reviews, and which I'm just delighted that
there's like 83 of them right now, but one of the things
that sort of actually make me think that we, you know, got
something, we didn't glorify him enough, in fact, some
people say that Brent is a villain.
It's like no, no, no, Brent is just trying to do what Brent
needs to do, right? And so I think in the ideal, especially
for IT operations, the people who are doing the majority of
the work, you know, the line contributors, the effects that
they will feel is that they're spending more time doing
creative work, improving the input environment, improving
the deployment mechanism and doing less time firefighting.
-25-
Interview: Calvin Powers, with Gene King, Ann Marie Fred and Michael Elder
Right? I mean, I think one of the most spiritually damaging
things we can actually do to people, to other people and
ourselves as human beings, is, you know, put them in a
system where they feel trapped, in a system that preordains
failure.
So when you're in IT operation and you have to live with the
downstream consequences of decisions made upstream of you
year after year after year that cause carnage and mayhem in
the production environment, you know, that make you work on
weekends and make every deployment, you know, sick for
weeks, for year after year, I mean, and you know, bring back
problems to the family.
I think these are the people whose lives will be most
improved by things like DevOps, because now we can actually
embed them into development, just like Ann Marie said, and
actually help change the outcomes and change the mindsets
and change how we make decisions.
And think on the development side, you know, it means that
we have to share the custodianship of the production
environment and the health of the deployment pipeline. And
I think that is great, because when we work on those things,
that's what actually allows us to speed up the tempo of
development and spend less time on rework and firefighting
-26-
Interview: Calvin Powers, with Gene King, Ann Marie Fred and Michael Elder
even in development.
POWERS: Very good. Now, one of our colleagues who
couldn't join us expressed to me in some e-mail that, you
know, just one tender criticism that maybe the book was a
little bit too operations centric and maybe quite didn't
give development teams their fair shake sometimes. I think,
Michael, you had some thoughts along those lines as well.
Can you kind of voice that concern for us?
ELDER: Sure. So, I think that the book makes a great
dramatic effect around the challenges of a development team
with zero discipline. But I think that if you look at some
organizations that have already adopted agile development
practices, there's already a feedback loop with continuous
integration, maybe even hopefully automated testing, static
analysis, et cetera, but that feedback loop hasn't yet been
extended into the operations process.
And so for some organizations, like the one described in the
book, you have to bring process and agility and feedback
loops even to the development process within itself and then
kind of carry it forward, and others where you have some
sense of feedback loop there but the handoffs are still very
manual, very error prone, right, there's no way that you're
automating the complete deployment into the system, it seems
like that's really an organization that still needs to
-27-
Interview: Calvin Powers, with Gene King, Ann Marie Fred and Michael Elder
extend it but where perhaps development is not the root of
all evil in the universe.
So I guess the question that we had was whether or not you
really feel from the vantage point that development tends to
be that cause because it is so feature driven, so, you know,
sort of a whiplash of the business side, right, bringing in
this capability of maybe yesterday we've got to go do it, or
whether that was just sort of a dramatic effect to make a
point.
KIM: You know, it's funny, well, I'll make two
observations. One is in my experience, and by the way, I
was a former developer. My first job was actually a QA
engineer for Sun Microsystems for the File System test
suite, so I have a lot of compassion for development.
It's been my experience that when there have been agile
processes in place and when there have been even continuous
integration, one of the things they don't do is actually
test in an environment that resembles production.
And so, they go do all this work, you know, to put
continuous integration practices in place, and yet they're
missing sort of the foundational piece that allows them to
actually, well, in some ways, and the worst case, right,
they're testing just a code and they never have the
-28-
Interview: Calvin Powers, with Gene King, Ann Marie Fred and Michael Elder
environment, you know, in the test plan.
And I think in my experience is that's at least 80 percent
of the organization. And so the notion wasn't to vilify
anybody, especially not development, because of course, some
of my best friends are developers.
[LAUGHTER]
It was just to show how much better it can be, right, if we
can just have development and operations share custodianship
around both the code and the environment. I guess what I
thought you were going to go with is like, boy, the kind of
comment that I've gotten most is like, wow, you guys are
sure hard on the information security person, right?
He...they sort of came off as, I don't want to say like a
buffoon, right, but I mean, he was a person who was
ostracized from the organization, marginalized. And Calvin,
when you and I were at AppSec USA, right, I mean, I think
there were a lot of chuckles in the room as kind of people
would recognize, you know, people that we knew, you know,
information security practitioners who everyone else hides
from because, you know, you invite this person to the
meeting, they're guaranteed to slow things down.
POWERS: That's right. We're the ones that are always
shaking our fists saying no, no, no, yes.
-29-
Interview: Calvin Powers, with Gene King, Ann Marie Fred and Michael Elder
KIM: Right. We were on this moral crusade, right,
that [should tell you] why we can't do things.
POWERS: That's right, which is exactly why we need to
get more risk based so that we can slide into this mode of
thinking, this continuous delivery mode of thinking. Well,
the way...the way I think that affects us is we need to
understand the risks and take them where necessary and
integrate our security operations into that continuous
delivery.
KIM: Right. And one of my biggest fears is that
information security will be the people saying, over my dead
body, right? We will never move to fast-flow features or
DevOps practices, you know, while I'm here. Right? Which
might be a career-jeopardizing move.
But I'm hoping that the book will actually show how
information security actually has some of the most to gain,
you know, by DevOps practices because we can now find and
fix issues faster. And you know, the fact that we have
short cycle times means that we can actually sort of
integrate more work, you know, we can actually inject more
work into the system and not only produce stability but also
improve security as well.
-30-
Interview: Calvin Powers, with Gene King, Ann Marie Fred and Michael Elder
FRED: Well, yes, I like the fact that you pointed out
that by using infrastructure as code and putting that into
source control now you have sort of an audit trail of what
exactly has been done to the system, right?
So that's a contribution from the development teams to
operations, hey, look at this, you know, now you have a
perfect way of tracking exactly what changed, who did it,
and also systems...you know, you can have systems where you
just reapply your deployment over and over again so if
somebody sneaks a change in it will just get wiped out in a
few minutes anyway. So I think it's nice to show that the
continuous delivery can actually improve your security and
audit posture also.
POWERS: That's interesting.
KIM: Here, here. Yes.
ELDER: And, in fact, I like the point that was made
when, I think it was when they were talking about deploying
more quickly using a cloud infrastructure and Brent made the
comment about, I've already got my own local tools that kind
of help me, right, I think a big part of what Ann Marie is
talking about, being able to pool some of that content that
Brent's created and help it benefit the integration at large
helps decouple Brent from being on the critical path of
everything, because you codify his tribal knowledge into a
-31-
Interview: Calvin Powers, with Gene King, Ann Marie Fred and Michael Elder
format that's more consumable for everyone else.
KIM: Absolutely. Well, and by the way, just, I know
this is...it seems some what germane, yeah, and so our goal
is not to vilify anybody except for one person, the VP of
marketing, right?
[LAUGHTER]
KIM: It was actually, we agonized about that for
months, is, you know, is she really a villain or is she also
sort of trapped in behaviors that are a side effect of the
way she's managed.
And essentially what we found is like, you know, one of the
parts of storytelling, and, you know, especially when you're
sort of trying to tell the hero's journey, you actually do
need a villain. And so, you know, after much sort of
agonizing, we said all right, you know, we've got to have a
villain, and the villain is her, so...
ELDER: Maybe that's your strategy, though, right, a
common enemy unites, you get dev and ops together to hate
one of the other lines of businesses on the critical paths
to delivering value to customers.
POWERS: Oh, Lord, have mercy. All right. On that fine
note, I think we're going to have to wrap this up. I want
-32-
Interview: Calvin Powers, with Gene King, Ann Marie Fred and Michael Elder
to say a thanks to my SMEs. Ann Marie, thank you so much
for joining us. Any closing remarks or plugs you want to
get in?
FRED: Oh, no. Just thank you very much for the book.
And yes, I'm personally going to go learn more about lean
myself, so.
POWERS: Terrific. You can all read her on the
Enterprise DevOps Blog on developerWorks, we'll have a link
on the video. And Michael, thanks to you also, and thanks
for your excellent questions and contributions. Any...?
ELDER: I will, if I can, take a quick opportunity,
take a look at what we have on jazz.net around SmartCloud
Continuous Delivery. You know, as Ann Marie pointed out, we
do have the DevOps blog as well. We very much love to
engage our community. And so, take a look, see what's out
there, and let us know what we can do to improve it or make
it better.
POWERS: Okay. And once again, that's jazz.net
SmartCloud Delivery, was that right?
ELDER: SmartCloud Continuous Delivery, that's correct.
KIM: I mean, I'll even amplify that. That's great
for developers and IT operations. I think continuous
-33-
Interview: Calvin Powers, with Gene King, Ann Marie Fred and Michael Elder
integration, continuous deployment practices are some of the
most important prerequisite skills to get sort of the
outcomes that we want when we talk about DevOps.
POWERS: All right. And once again, the book is The
Phoenix Project: A Novel about IT, DevOps and Helping Your
Business Win. And I would be remiss if I didn't point out
that it's co-authored also with Kevin Behr and George
Spafford, the one and only George Spafford. So, kudos on
that. Gene, thanks for joining us. Any closing remarks or
plugs you want to get in?
KIM: Oh, no. Thank you so much. And if...I'll send
together for show notes some resources about some of the
principles and underpinning theories that went into the book
as well as certainly a link to the set of articles that we
did for...that we worked on together, Calvin.
[END OF SEGMENT]
-34-