69
Don’t let this happen to you! Cloud, complexity and drift James Urquhart @jamesurquhart http://gigaom.com/cloud
Don’t let this happen to you!Cloud, complexity and drift
James Urquhart@jamesurquhart http://gigaom.com/cloud
May 6, 2010
”[A] large fundamental trader chose to execute [a $4.1B] sell program via an automated execution algorithm ('Sell Algorithm')."
- Findings Regarding The Market Events of May 6, 2010http://www.sec.gov/news/studies/2010/marketevents-report.pdf
"...the Sell Algorithm…executed the sell program extremely rapidly in just 20 minutes.”
The market responded, and trading volume increased…
"... [The Sell Algorithm] responded to the increased volume by increasing the rate at which it was feeding the orders into the market."
- Findings Regarding The Market Events of May 6, 2010http://www.sec.gov/news/studies/2010/marketevents-report.pdf
Quick!What was the root cause?
Market AAutomatic
Trading Algorithm 1
Market AAutomatic
Trading Algorithm 1
Automatic Trading
Algorithm 2
Automatic Trading
Algorithm 3
Automatic Trading
Algorithm 4
Market AAutomatic
Trading Algorithm 1
Automatic Trading
Algorithm 2
Automatic Trading
Algorithm 3
Automatic Trading
Algorithm 4
Market B
Market AAutomatic
Trading Algorithm 1
Automatic Trading
Algorithm 2
Automatic Trading
Algorithm 3
Automatic Trading
Algorithm 4
Market B
Automatic Trading
Algorithm 5
Automatic Trading
Algorithm 6
Market AAutomatic
Trading Algorithm 1
Automatic Trading
Algorithm 2
Automatic Trading
Algorithm 3
Automatic Trading
Algorithm 4
Market B
Automatic Trading
Algorithm 5
Automatic Trading
Algorithm 6
Automatic Trading
Algorithm N
Automatic Trading
Algorithm N Market A
Automatic Trading
Algorithm 2
Automatic Trading
Algorithm 3
Automatic Trading
Algorithm 4
Market B
Automatic Trading
Algorithm 5
Automatic Trading
Algorithm 6
Automatic Trading
Algorithm N
“Root cause” is not an answer—it’s a clue.
"May 6 was…an important reminder of the interconnectedness of our derivatives and securities markets, particularly with respect to index products."
- Findings Regarding The Market Events of May 6, 2010http://www.sec.gov/news/studies/2010/marketevents-report.pdf
Interconnectedness
The Internet is about interconnectedness.
APIs are about interconnectedness.
“The cloud” is about interconnectedness.
Both cloud computing and stock markets are
complex adaptive systems.
COMPLEX?!?
What areComplex Adaptive
Systems?
A large number of individual agents
Agents
State
Rules
OtherAgents
OtherAgents
Information/Actions
Information/Actions
Learning
A large number of individual agents
+ dynamic interactions between agents
Dynamic Interaction
From NetLogo 4.1.3
A large number of individual agents
+ dynamic interactions between agents
+ rules for reacting to/interacting with other agents
Rules
Equals
A system that:
demonstrates emergent behavior,
Emergent Behavior
A system that:
demonstrates emergent behavior,
can be modeled,
Models
Time
Flocking model from NetLogo 4.1.3http://ccl.northwestern.edu/netlogo/
A system that:
demonstrates emergent behavior,
can be modeled,
but
makes precise prediction of future behavior impossible.
Unp
redi
ctab
ility
Unpredictability
Unpredictability
Unp
redi
ctab
ility
So…what can you do?
EMBRACE COMPLEXITY
Never!
EMBRACE COMPLEXITY
SYSTEMS
Embrace Systems Thinking
Practical advice?
Do your homework!
Design for resilience
“The goal of producing a maximum sustained yield may result in a more stable system of reduced resilience.”
“Command and Control and the Pathology of Natural Resource Management”, C. S. Holling and Gary K. Meffehttp://landscape.forest.wisc.edu/courses/Landscape565spr01/Holling_Meffe1996.pdf
“[T]he resilience-stability tradeoff is more than just a simple transformation in distribution. …[A]gents adapt to a prolonged period of stability in such a manner that the system cannot ‘withstand even modest adverse shocks.’”
“The Euro and the Resilience-Stability Tradeoff”, Ashwin Parameswaranhttp://www.macroresilience.com/2011/11/14/the-euro-and-the-resilience-stability-tradeoff/
Focus on relationships
Automatic Trading
Algorithm N
Automatic Trading
Algorithm N Market A
Automatic Trading
Algorithm 2
Automatic Trading
Algorithm 3
Automatic Trading
Algorithm 4
Market B
Automatic Trading
Algorithm 5
Automatic Trading
Algorithm 6
Automatic Trading
Algorithm N
Automatic Trading
Algorithm N
Automatic Trading
Algorithm N Market A
Automatic Trading
Algorithm 2
Automatic Trading
Algorithm 3
Automatic Trading
Algorithm 4
Market B
Automatic Trading
Algorithm 5
Automatic Trading
Algorithm 6
Automatic Trading
Algorithm N
Circuit Breaker
http://techblog.netflix.com/2011/12/making-netflix-api-more-resilient.html
Avoid Drift
Drift
Avoiding Drift:Test production
Release the monkeys!!!• Failure• Degredation• Usage
Avoiding Drift:The agent is not the
system
App-centric• Monitor each app
separately• Dig for root cause• Attempt to stabilize
System-centric• Monitor system at many
levels, and use as feedback• Search for system weakness• Focus on resilience
Avoiding Drift:
YOUare part of the system
Don’t let this happen to you!
Cloud, complexity and drift
![ഠto happen and what actually does happen.]It's a black fly in your Chardonnay . It's a death row pardon two minutes too late . And isn't it ironic... don't you think . Activity D\爀屲Now](https://static.documents.pub/doc/80x56/5f0532a57e708231d411c37f/ato-happen-and-what-actually-does-happen-its-a-black-fly-in-your-chardonnay.jpg)
