Hello, my name is Tristan Slominski and today I will talk to you about scaling an

organization from first principles

Now… this presentation is going to feel a little bit like this… that is… that’s what I’m going to look like to you… It will feel like I’m going a little bit all over the place, but I promise to tie it all together before the end. It is an artifact that a lot of concepts are pulled together from multiple disciplines that usually don’t overlap much...

Let’s get started with an excerpt from Mark Burgess’ “In Search of Certainty: The Science of Our Information Infrastructure”... Imagine a scene in a movie. A quiet night on the ocean, on the deck of a magnificent ship, sailing dreamily into destiny. Moonlight reflects in a calm pond that stretches off into the distance, waves lap serenely against the bow of the ship, and – had there been crickets on the ocean, we would have heard that reassuring purring of the night to calm the senses. The captain, replete with perfectly adjusted uniform, comes up to the night helmsman and asks: “How goes it, sailor?” To which the sailor replies: “No problem. All’s quiet, sir. Making a small course correction. Everything’s shipshape and under control.” At this moment, the soundtrack stirs, swelling into darker tones, because we know that those famous last words are surely a sign of trouble in any Hollywood script. At that very moment, the camera seems to dive into the helmsman’s body, swimming frantically along his arteries with his bloodstream to a cavernous opening, where we view a deadly parasite within him that will kill him within the hour. Then the camera pulls back of him and pans out, rising above the ship, up into the air to an altitude at which the clear, still pond of the ocean seems to

freckle and soon becomes obscured by clouds. The calm sea, it turns out, is just the trough of a massive wave that, miles away, reaches up to ten times the height of the ship, and is racing across the planet with imminent destruction written all over it. As we rise up, and zoom out of the detail, we see the edge of a massive storm, swirling with fierce intensity and wreaking havoc on what is now a hair’s breadth away on the screen. And then, pulling even farther out, just beyond the edge of the planet, is a swarm of meteorites, firing down onto the human realm, one of which is the size of Long Island (it’s always Long Island), and will soon wipe out all life on Earth. Picking up speed now, the camera zooms back and we see the solar system spinning around a fiery sun, and see stars and galaxies and we return to a calm serenity, where detail is mere shades of colour on a simple black canvas. Then the entire universe is swallowed into a black hole. Shipshape and under control? Burgess, Mark. In Search of Certainty: The Science of Our Information Infrastructure (pp. 11-12). O'Reilly Media. Kindle Edition.

The concept that the story is intended to convey is the concept of scale.

Throughout this presentation, I want you to try to keep in your mind the concept of scale. Many small components coming together... http://upload.wikimedia.org/wikipedia/commons/9/9e/Graphen.jpg

… to create more complex components... http://www.molsoft.com/images/antibiotic_surface.jpg

… that create other components with defined interfaces ... http://31.media.tumblr.com/aa9e31b892c0c38876858f9662f1d938/tumblr_mia8seamuN1qbh26io3_1280.jpg

… that together define many small components ... http://www.nisenet.org/sites/default/files/red_blood_cells_nise.jpg

… that come together to create more complex components ... http://www.joelertola.com/grfx/cap/cap_lg.jpg

… with well defined interfaces ... http://blog.stickemarchery.com/wp-content/uploads/2011/08/DSC00926.jpg

… that come together … http://upload.wikimedia.org/wikipedia/commons/9/98/Lesotho_mountain_village_(5285775857).jpg

…. to create components ... http://upload.wikimedia.org/wikipedia/commons/4/42/Corfu_town.jpg

… that come together ... http://learnenglish.com/blog/wp-content/uploads/2013/06/New-York-skyline_0.jpg

… to create components ... http://eoimages.gsfc.nasa.gov/images/imagerecords/79000/79800/dnb_united_states_lrg.jpg

… that combine in complex ways … http://gigaom2.files.wordpress.com/2010/12/163413_479288597199_9445547199_5658562_14158417_n.png

… to create components with well defined interfaces ... http://d1jqu7g1y74ds1.cloudfront.net/wp-content/uploads/2009/09/bluemarble.jpg

… etc etc ..

… and remember that while things are different at different scales….

… everything matters When you get a cold, the effects you feel are caused by viruses… at scales we couldn’t even imagine for most of human history.

So, how do we make sense of all this complexity?

One way to frame complexity is via something called the Cynefin framework… Simple: drop something Complicated: car engine; can take it apart into pieces and put it back together Chaos: trees exploding on a walk Complex: most of life; cannot take it apart into pieces and put it back together

Let’s focus on the differences between Complex and Complicated. This is the crucial distinction to understand. (click) Why can we predict some things whereas others can only be understood in hindsight?

Let’s say we want to take a picture of Pluto...

We do some math… we figure out which way we ought to go… and we launch a spaceship that travels for nine and half years, 3.26 billion miles and gets within 7,759 miles of Pluto and takes some pictures. (click) How do you even…? What condition makes such a feat possible?

A key observation to make is that the laws of physics don’t change on the time scale we care about

In other words, the constraints we are operating within remain fixed. You can probably then guess the nature of constraints in a complex system (click) … (click) due to agent actions

What is an example of a constraint changing due to agent action? A pursuit of a career changes our constraints (we pursue opportunities associated with that career over others). Attending this talk changed your constraints (you had to schedule things in order to attend). If you’re on-call, getting paged by a machine, changes our constraints (we now need to drop what we were doing and resolve an issue). Almost every action we take as people, changes some constraints in some way.

Let’s recap (click) We’ve talked about Scale, and that things are different at different scales, yet everything matters. (click) We’ve talked about Complexity, where cause and effect relationship is knowable only in hindsight, and where every action we take changes some constraints we operate with. Let’s now look at the effect of complexity on our ability to scale...

Imagine I have an item of work The length of this segment represents how long it takes to execute the work

Let’s add a worker It’ll take a single worker this long to finish the work

Now, let’s divide this work into four shards…

We can complete the work in ¼ of the time

This graph depicts this concept of linear scalability. That is, the more workers we have, the faster we can accomplish the work. The function X(N) (X of N) is the throughput given N workers. 𝛾 (gamma) in this case, is equal to 1.

Now, our model so far is missing an aspect of what it takes to do work.

The extra segments represent the additional time required for distributing and integrating the work shards. We can also think of this as contention, or queueing, i.e. waiting for a resource. In this case, the first worker can only distribute one work shard at a time and can only integrate one work shard at a time. Notice that this distribution and integration adds some time to accomplish the work.

Accounting for the contention/queuing/waiting aspect of the system, we end up with Amdahl’s Law. We added to the denominator of the equation. The component that represents contention, 𝜎 (sigma), is the fraction of the work that cannot be parallelized. Even with 1%, or 0.01 of the work being serialized, we see the diminished throughput.

But… waiting/queueing is not the only thing that impacts our ability to accomplish work. What happens when the workers need to coordinate with each other?

Each worker now makes other workers do extra work... This is bad (click) O(n^2) bad

This is the nature of coherence… the delay for data to become consistent (or coherent) by virtue of point-to-point exchange of data between resources that are distributed

This is the nature of coherence… the delay for data to become consistent (or coherent) by virtue of point-to-point exchange of data between resources that are distributed.

Accounting for coherence aspect of the system, we end up with the Universal Scalability Law. We added to the denominator of the equation. The component that represents coherence, 𝜅 (kappa), represents the percentage of the work due to crosstalk/coordination. With 𝜅 set as low as 1% or 0.01 we already see the point of diminishing returns. For work with 1% contention and 1% coordination, 10 workers is the point after which we get diminishing returns. For context, 5 minutes is 1% of an 8 hour work day. Raise your hand if your work consists of less than 5 minutes per day of waiting and less than 5 minutes per day of coordinating? I also want to highlight the range of a “two pizza” team, typically 7 +/- 5. So what? Coordination kills throughput … and not a lot of coordination… just a tiny bit is devastating

Let’s recap We’ve talked about Scale, and that things are different at different scales, yet everything matters. We’ve talked about Complexity, where cause and effect relationship is knowable only in hindsight, and where every action we take changes some constraints we operate with. We’ve talked about the Universal Scalability Law where we learned that coordination kills throughput. Let’s build an organization as if scalability, complexity, and universal scalability law mattered…

We start with a solo contributor… Scale: person, Complexity: contained within one mind; Contention: limited to one person doing the work; Coherence: maximum

We want to get more work done, so we grow our team… Scale: person Complexity: 1 feedback loop Contention: we can do more work with two people Coherence: pretty good (graph shows 1% of time spent coordinating)

We want to get more work done, so we grow our team… Scale: person Complexity: 10 feedback loops Contention: Lots of work can happen Coherence: pretty good, depends on % time spent coordinating (graph shows 1% of time spent coordinating)

We want to get more work done, so we grow our team… Scale: person Complexity: 45 feedback loops Contention: Lots of work could happen Coherence: 5 minutes per day of coordinating severely limits our throughput...

We split into two teams Scale: teams *scale shift* Complexity: 10 feedback loops *per team* Contention: almost full two team’s worth of work can happen Coherence: improved at people scale quite a bit; and negligible at two-team scale...

Let’s visualize the feedback loops between people… which correlates with complexity…

Let’s visualize the feedback loops between people… which correlates with complexity…

When we then split into two teams, what happens to the feedback loops is crucial. If we do not eliminate feedback loops when we split into two teams… we haven’t managed to address the coherence penalty brought on by the amount of coordination required to do the work.

By eliminating the feedback loops required between teams to deliver work… we increased each teams’ coherence (at the scale of people) Notice that while the arrow points from 10 to 5 people, the work throughput goes from 5 to 4… but we now have two teams… so work throughput is 2 x 4 = 8 … compared to previous 5. Also… I do want to point out that eliminating feedback loops here doesn’t mean never talking to each other. The implication is that for our daily work, we do not need to coordinate with people from other teams… you can still be friends.

Once the teams are coherent… (they went through forming - storming - norming) we then shift the scale from the scale of people onto the scale of teams…. And let’s add the feedback loop between the teams to the picture.. So, we have a feedback loop.. But what is it? What does it consist of?

It’s an API. This is why APIs are useful.

We’re almost to SLOs… but first, let’s go over what happens when the API is missing. Notice that the way we arrived at an API is by eliminating inter-team feedback loops between the people. Another name for an inter-team feedback loop between people is a conversation or interaction. If I don’t have an API to call, or the API is inadequate, I have to engage with another team member, say Ashley here, and coordinate. And we’re friends, and we’re helpful, so we’ll help each other out. Work got done. As a note, I do want to highlight that there are things happening at two different scales… team-to-team coordination overlayed on top of person-to-person coordination. Now, recall that coordination destroys throughput… and just a tiny bit of it is enough to destroy most of the benefits So when the two of us come to arrangement, Juan now wants to make some changes, and Ashley points out that I’m doing a specific thing per our arrangement, so Juan coordinates their changes with me… Meanwhile.. Ashley became the “go to” person for the thing… so I tell Brianna to talk to Ashley.. One thing leads to another… and slowly we are rebuilding the complexity we worked

so hard to contain.. Most importantly.. We lost the benefits of team scale and shifted work back to person scale… where things don’t scale as well due to coordination penalty. We are shifting our work from team scale to people scale and losing the scalability benefit of reduced coordination by reintroducing all the coordination. I will leave it for you to ponder where a JIRA ticket fits into this picture, and whether that makes for a good API.

We can now finally explain why SLOs are useful. Let’s assume we avoided all the traps of coordination and our teams and services are communicating only via APIs… What happens when I make an API call… … and get an error response

Well… I have to engage with another team member, say Ashley here, and coordinate. I tell Ashley that I’m getting errors, and I don’t want errors, and Ashley looks into the problem and tells me that is expected, or maybe not. And we’re friends, and we’re helpful, so we’ll help each other out. Work got done. So when the two of us come to arrangement, Juan now wants to make some changes, and Ashley points out that I have specific performance expectations per our arrangement, so Juan coordinates their changes with me… Meanwhile.. Ashley became the “go to” person for the thing… so I tell Brianna to talk to Ashley.. One thing leads to another… and slowly we are rebuilding the complexity we worked so hard to contain.. We are shifting our work from team scale to people scale and losing the scalability benefit of reduced coordination by reintroducing all the coordination. This, despite having a well defined and usable API.

So… where we have an API in order to restrict inter-team coordination required between individuals regarding the functionality between the teams, thus allowing our organization to shift towards the team scale... We have SLOs in order to restrict inter-team coordination required between individuals regarding the performance between teams, thus allowing our organization to shift towards the team scale… But wait.. There’s more...

Recall the complex vs. complicated framing… Recall that in a complicated domain, we are able to plan for the future, whereas in a complex domain we must be in a constant feedback loop with our environment in order to react appropriately to changing constraints... Recall the relationship between the domains and the rate of change of constraints….

When we have teams with APIs and SLOs… the APIs and SLOs serve as a way to reduce the rate of change of constraints that the teams have to deal with. Think about what a stable API and a stable SLO offers… In essence, it means that while within my team, I am dealing with the (click) Complexity of everything that happens… I can treat my dependencies behind the API/SLO abstraction as (click) Complicated… because… There are people on those teams, dealing with their Complexity… and they can treat all their dependencies behind the API/SLO abstraction as Complicated

So, here is how APIs and SLOs enable us to scale an organization..

So far… I’ve been trying to get across the important difference between complex and complicated domains… Now, let’s take a closer look at complexity in our business context so that we get a better idea whether or not our APIs and SLOs are well formed and achieved the goal of actually encapsulating complexity…

To get at the fine grained structure of complexity in the business context, we’ll use Wardley Maps...

This is a Wardley Map… In particular, this is an example map for an online photo service from 2005.

Let’s take a moment to familiarize ourselves with the various aspects of a Wardley Map so that you’ll be able to get the gist of the maps I show next…. The map is visual and context specific i.e. it is unique to that line of business containing the components that influence it at that moment in time. The map has an anchor, which is the customer. The position of components in the map are shown relative to that Customer on a value chain, represented by the vertical-axis. Each component needs the component below it. The closer to the Customer and the higher up on the vertical-axis, the more visible the component to the Customer. Further down on the vertical-axis, the less visible the component to the customer. For example, the Customer cares about online photo storage, which requires underlying components like Data Centre and Power, those are further away from the Customer and hence less visible. The components of the map also have a stage of evolution, the horizontal-axis. Genesis: This represents the unique, the very rare, the uncertain, the constantly changing and the newly discovered. Our focus is on exploration.

Custom built: This represents the very uncommon and that which we are still learning about. It is individually made and tailored for a specific environment. It is bespoke. It frequently changes. It is an artisan skill. You wouldn’t expect to see two of these that are the same. Our focus is on learning and our craft. Product (including rental): This represent the increasingly common, the manufactured through a repeatable process, the more defined, the better understood. Change becomes slower here. Whilst there exists differentiation particularly in the early stages there is increasing stability and sameness. You will often see many of the same product. Our focus is on refining and improving. Commodity (including utility): This represents scale and volume operations of production, the highly standardised, the defined, the fixed, the undifferentiated, the fit for a specific known purpose and repetition, repetition and more repetition. Our focus is on ruthless removal of deviation, on industrialisation, and operational efficiency. With time we become habituated to the act, it is increasingly less visible and we often forget it’s even there. All the components on the map are moving from left to right driven by supply and demand competition. The components evolve.

There is a lot to Wardley Mapping that we aren’t going to begin to cover here, but one aspect I want to highlight is the aspect of movement, the fact that components change as they evolve towards Commodity/Utility. Take a note of the characteristics at the Genesis side of the evolution axis and compare them to the characteristics on the Commodity side. On the left we have “uncertain”, “unpredictable”, “poorly understood”, “competitive advantage”... and on the right “ordered”, “known”, “stable”, “obvious”, “operational efficiency”, “cost of doing business”. (click) If we view these through our previous framing, we can see that components go from being thought of as Complex, to eventually being thought of as being Complicated.

If we go back to the idea that creating APIs and SLOs contains complexity and allows us to scale… we see that those same techniques help us provide stability and enable us to be viewed as complicated by the users of our APIs and consumers of our SLOs… these techniques allow us to shift our service or component toward a commodity/utility. But why does a shift toward commodity/utility matter?

Shifting towards commodity matters because efficiency brought on by commodity/utility enables innovation and creation of new sources of worth. (click) A shift toward commodity stabilizes the system and freezes the constraints. This, in turn, allows others to generate new complex things on top of stable, complicated, commodities/utilities.

In turn, those complex things become stable, and we invent new complex things on top of those, and so on… thus allowing us to survive in the business environment while playing the infinite game.

So, before we said that this is how APIs and SLOs enable us to scale an organization..

But now, you hopefully have a better idea of what this nebulous “accomplish more” means...

… commoditization allows us to generate new complex sources of worth… … and that’s sort of why we’re all here… … that’s how we can keep playing the infinite game...

Complete graph icons by Koko90 - Own work, CC BY-SA 3.0,

https://commons.wikimedia.org/w/index.php?curid=10947367

Questions...