@helenaedelson

Disorder & Tolerance in Distributed Systems at Scale

Rethinking intelligent resilient systems

Helena Edelson, Scale By The Bay 2017

@helenaedelson

Seen In The Wild

Committer/Contributor FiloDB, Akka, Spark Cassandra Connector, Kafka Connect Cassandra, Spring Integration

Helena Edelson

twitter.com/helenaedelson

Program Committee Member Kafka Summit 2018Reactive Summit 2016-2017

Speaker Kafka Summit, Spark Summit (EU, NYC), Strata (NYC, SJ), QCon SF, Scala Days (EU, NYC), Reactive Summit (’16, ’17), Philly ETE, Scale by the Bay!

linkedin.com/in/helenaedelson

@helenaedelson

• Interdisciplinary look at how complex adaptive systems apply to distributed systems and information engineering

• Systems, intelligence and theories

• Entropy, Events and Time

• Rethinking adaptive systems, complexity and resilience

Different Approaches

@helenaedelson

Inspired By

• My scientific research before working in tech

• What I've noticed in the industry over almost two decades

• Questioning how we approach distributed systems, balance and disorder

Finding better ways to handle system dynamics

• Creating models to predict system dynamics

• Re-engineer energy flows in biological systems

• Slow the rate of entropy in those systems

@helenaedelson

– Albert Einstein

“Problems cannot be solved with the same mind set that created them.”

@helenaedelson

Intelligent Systems

@helenaedelson

It's All About InformationData: much of what our systems support and transport

@helenaedelson

@helenaedelson

sys·tem

• An entity comprised of interdependent elements and subsystems

• More than the sum of its parts

• Has feedback loops

• Defined by its distinguishing edges

In this talk we refer to open systems

@helenaedelson

Systems Theory

• Discovering how elements of a system and its sub-systems interact to produce given end states

• To understand a system's dynamics• Changing one part affects others in the system • Many systems-related theories developed out of this

Interdisciplinary study of systems

@helenaedelson

Bertalanffy proposed that Systems Theory needed a much broader, unified approach

• Transcending technical problems

• Applicable to all scientific study (biology, physics...)

General System TheoryWas a new paradigm for scientific inquiry

@helenaedelson

Complex Adaptive Systems Theory

• Used to model an array different systems

• Complex, Non-Linear Systems: how order emerges, e.g. in neural networks, galaxies, ecosystems

• Self-organization - suggests living systems can migrate to a dynamic state, the ”edge of chaos”

- This discipline suggests living systems migrate to a state of dynamic stability they call the "edge of chaos" or balance point.

Complexity Theory

@helenaedelson

Distributed Systems

• With increasing scale comes increased complexity and potential for disorder

• The more moving parts in a system, the more things that can fail

• In biological systems, the greater the diversity and/or complexity, the greater the overall resilience

The larger the scale, the greater potential to fail

@helenaedelson

The Butterly EffectWeather prediction: small causes can have larger effects

@helenaedelson

Ensemble ForecastingRange of possible future states

@helenaedelson

Ensemble ForecastingWildfire prediction: a range of possible future states,

differing initial conditions

@helenaedelson

Destruction as Transformative ForceLaying the foundation for next state of energy

end state = regeneration

@helenaedelson

Entropy, Events And TimeOrder and disorder, time as events

@helenaedelson

Second Law of Thermodynamics

• The law from physics stating that entropy increases

• Measures the degree of disorder of a system

• The increase in entropy accounts for the irreversibility of natural processes, and the asymmetry between future and past

Entropy

@helenaedelson

Entropy And The Arrow Of Time

"If given complete knowledge of the universe for two instances of time, how would you solve which instance happened first?

Order DisorderTime

Calculate the entropy of the two snapshots. The one with lower entropy was first."- Muller, Richard A, The Physics of Time

@helenaedelson

Future Light Cone

"If the sun were to cease to shine at this very moment, it would not affect things on earth at the present time because they would be in the elsewhere of the event when the sun went out."

- Stephen Hawking, A Brief History of Time, 1988

Stephen Hawking, A Brief History of Time

@helenaedelsonStephen Hawking, A Brief History of Time

• Events lie in the future light cone everywhere that is not its origin

• When we look at the universe we are seeing the past

@helenaedelson

Time As Derivative Of Events?

Events are sequences of things happening in time

OR

Time is a sequence of events

@helenaedelson

–Anthony Aguirre

“Maybe it’s more accurate to say that time flows as events happen. The flowing of time

or passage of time, is events.”

@helenaedelson

NowThe sense that time moves forward, in the continual

creation of new nows

@helenaedelson

Biological SystemsIntelligent, Adaptive, Self-Organizing Systems

@helenaedelson

We Are All HostsVirus as champion of adaptation and co-evolution

@helenaedelson

The Immune System

• Exhibits a highly distributed, adaptive and self-organizing behavior

• Is a self-programming system

• Infinite ability to re-program itself to destroy threatening microbes

• Is a self-learning system

• Learns in parallel to fight the many forms of virus

@helenaedelson

Complexity & ResiliencyFrom systems theory

@helenaedelson

Domino Effect• Change of one can trigger

change in others

• Genesis event

• As elements of the system are effected, they generate more events

• E.g. cascading failure

@helenaedelson

Evolution & Complexity At The EdgeThriving complex systems at transition zones

@helenaedelson

Self-Organization

• We tend to assume that organization and order need to be imposed by some external force.

• Self-organization is the idea that this type of global organization can instead be the result of local interactions.

@helenaedelson

Musk Oxen in the arctic organize to form a circle around the youngPeer to Peer Organization

@helenaedelson

Self-Organization: Emergenceschooling, swarming, herding

@helenaedelson

Emergence

Ant colonies are governed by very simple rules, and only local interactions. Through combined activities, generate colonies that

• Exhibit complex structures and behavior

• Far exceed intelligence or capability of the individual

• Decentralized structure to self-organizing systems

• Organization is distributed over the whole system

• All parts contribute equally

Case Study

@helenaedelson

Traditional centralized organization is relatively static model.Self-organization is dynamic, with autonomous members densely interacting locally.

Economies of scale

@helenaedelson

Cyclic, Predictable Patterns & Resilience

Biological systems have natural feedback loops and strategies that enable resilience to fluctuation.

The Three Rs

• Replication

• Regeneration

• Rebalance

@helenaedelson

Self-Organizing PatternsMigration

@helenaedelson

Annual Pattern of Movement

Arctic Tern

• Longest migration on earth

• Pole to pole and back every year

@helenaedelson

Daily Pattern of Movement

Arctic Wolves • Top of their food chain• Operate in packs, 30+• Pack roams its territory daily• Travel 40-100 miles per day• Follows herd food sources

annually in their migration

@helenaedelson

Predictable patterns in time and space that are changed and cause change

sea·sons

@helenaedelson

Planetary Orbit and Axial TiltChanges cascade to all elements in all systems

@helenaedelson

Resilient Systems & DiversityVariety of entities makes the systems more effective at absorbing change.

and variations in its environment.

@helenaedelson

Role Niche• Organisms role in an

ecosystem• The environment of the entity • What it consumes• How it interacts with other

elements or entities

• Entities role in a system• Data ingestion• Functions in the system• How it interacts with other

elements or entities

If the number of entities performing a necessary function in a system decrease, the system can fall into imbalance.

@helenaedelson

– John Muir

“When we try to pick out anything by itself, we find it hitched to everything else in the Universe.”

@helenaedelson

Tropic Cascade

A process which starts at the top of the system or meta-system hierarchy, eventually affecting all the way down to the base.

@helenaedelson

– Stephen Hawking

“It is a matter of common experience that disorder will tend to increase if things are left to themselves.”

@helenaedelson

Tropic Cascade Case StudyA complex system in constant change

In 1926 the last wolf in Yellowstone NP in the US was eliminated.

By 1994 the elk population grew to roughly 19,000.

@helenaedelson

Elimination of the wolves caused a cascade of changes through the entire ecosystem.

With no natural predator, Elk consumed most of their food resources.

Tropic Cascade Case StudyA complex system in constant change

@helenaedelson

Destabilization

As elk increased

• Berries for bear food supply decreased

• Bear population fell to Endangered Species levels

• The coyote population increased to partially fill the niche left by the wolves

• Tree and plant hight and numbers decreased dramatically

Absence of top predator altered the entire system

@helenaedelson

Reintroduction

• In 1995 14 grey wolves from Canada were introduced to Yellowstone, after being absent for over 60 years

• A year later 17 wolves were introduced

• By December, 2001 their population had grown to 132

Of entities performing the primary regulating role

@helenaedelson

Adaptation & Predatory PressurePredatory pressure keeps prey on the move so they

don't use up resources in one area

@helenaedelson

Regeneration

Elk started to avoid parts of the park where they were more exposed for the wolves to hunt.

• Forests of aspen and willow began growing back• As bushes and grasses grew back, there were more berries• The diversity and number of birds started increasing

@helenaedelson

RepopulationTrees started to grow taller again as the elk population decreased.

• Beaver, previously extinct in the region, returned• The dams beavers built provided habitat for otters and

other animals and reptiles• Wolves hunted the coyote, decreasing their population 50%• The numbers of rabbits and mice were able to grow back• Which brought more red foxes, weasels, badgers

• The bald eagle and hawk populations grew

@helenaedelson

The Bison population began to grow back.Large Mammal Populations Rebalanced

@helenaedelson

Diversity RebalancedLarge mammals can not thrive unless diversity in

their system is also balanced

@helenaedelson

Rebalance

With the rebalancing of predator / prey, the populations of many other species were again able to rebalance.

• The vegetation along rivers and lakes returned• Erosion decreased• Which changed the shape of the rivers• River banks stabilized, channels narrowed• More pools of water formed• Increasing habitat for water birds and reptiles

@helenaedelson

One Rolecan change the entire topology

@helenaedelson

– Stephen Hawking

“It is a matter of common experience that disorder will tend to increase if things are left to themselves.”

Self-Balancing Systems

@helenaedelson

Innovationassembly line versus research

@helenaedelson

ResearchThere was a time when companies weren’t afraid to invest in basic science.

Companies still invest heavily in innovation, but the focus is practical applications rather than basic science.

Research and development has become “less R, more D” - Prof. Ashish Arora, economics of technology and technical change

@helenaedelson

Rate Of Innovation

• Why is information technology seemingly behind technology in scientific fields such as astrophysics, particle physics, molecular biology and behavioral neuroscience?

• They have made phenomenal gains but the compute systems that network and manage them, and also capture, process, store and query those system's data has not seen the same speed in innovation.

@helenaedelson

Be ExperimentalGather real data vs

assumption planning without proof

@helenaedelson

– Kip S. Thorne, Nobel Prize in Physics, 2017

“Huge discoveries are really the result of giant collaborations”

@helenaedelson

Thank You!@helenaedelson