Date post: | 07-Aug-2015 |
Category: |
Software |
Upload: | jon-schneider |
View: | 90 times |
Download: | 3 times |
Freedom, Responsibility, and Dependencies
JON SCHNEIDERSenior Software Engineer – Build Tools at Netflix@jon_k_schneider
JFrog SwampUP – May 2015
Responsible people thrive on freedomand are worthy of freedom
Why do most companies curtail freedom and become bureaucratic as they grow?
Desire for Bigger Positive Impact Creates Growth
Growth
Growth Increases Complexity
Complexity
Growth Also Often Shrinks
Talent Density
% High Performance Employees
Growth + Complexity
Chaos Emerges
% High Performance Employees
Complexity
CHAOS
Process Emerges to Stop the Chaos
Process
Process-focus Drives More Talent Out
% High Performance Employees
Process
Then the market shifts…
The Key: Increase Talent Density Faster Than Complexity Grows
% High Performance Employees
Business Complexity
The Binary Dependency Choice
[1] Much like Scala itself, the SBT logo has gone through at least 3 iterations
[1]
Freedom begets responsibility?
My freedom
The responsibility costof my freedom to othersin my organization
“The correct amount of constraint enables creativity more than total freedom does”
Freedom begets responsibility?
My freedom
The responsibility costof my freedom to othersin my organization
Our role is to
curtail this effect
“The correct amount of constraint enables creativity more than total freedom does”
Build Tools Focus: Nebula
REDUCEBOILERPLATE
ENRICHARTIFACTMETADATA
IDENTIFY PROJECT OWNERS
COLLECT BUILD METRICS
Our gaze has shifted to dependency management
Upstream vs. DownstreamDependencies
Astrid Downstreams
Version
ocp~1
b~5
d~7
b~6
c~5
c~6
ocp~2
version=1
version=1 version=2
version=2
Skew
ocp~1
b~1 b~2 b~3 b~4
c~1 c~2
skew=0
skew=0
skew=0skew=1
skew=3
Adoption Rate
lam
bda =
1
lam
bda =
4
greater lambda = faster adoption
Degrees of Separation
ocp~1
b~1
d~1
c~1
degrees=1
degrees=1
degrees=2
Astrid Upstreams
Dependency Insight: Astrid
Release Notes
Dependency
Resolution
Upstream
Downstream
Event Store
Read Models
Version conflict resolution is the hardest problem we face
The Conflict Resolution Problem
app
guava:14
guava:14
guava:14
unit tests prove the accuracy of these systems relative to guava 14
The Conflict Resolution Problem
app
guava:16
guava:14
upgrade to a later version
guava:14
The Conflict Resolution Problem
app
guava:16
guava:16
causes evictions
guava:14
guava:16
guava:14
The Conflict Resolution Problem
app
guava:16
guava:16
these code paths need to be tested for correctness
guava:14
guava:16
guava:14
Consequences are starting to get serious…
3 Classes of Solutions
Stability Avoidance Modularization
non-intrusive intrusive
few many
scope of problems solved
intrusiveness
earlier recent
recency as a solution at Netflix
Stability solutions are stopgap measures that postpone resolving conflict
Nebula Dependency Locking is our main stability solution
Dependency Locking
1
2
3
./gradlew generateLock saveLock
./gradlew test
./gradlew commitLock
if tests pass,
Dependency Locking
{ "com.google.guava:guava": { "locked": "14.0", "requested": "14.0" }, "org.foo:foo": { "locked": "1.0", "requested": "1.0" }}
Extra artifact produced that is later used to add forces on dependencies
3 Classes of Solutions
Stability Avoidance Modularization
non-intrusive intrusive
few many
intrusiveness
earlier recent
recency as a solution at Netflix
scope of problems solved
What if everybody built more frequently?
3 Classes of Solutions
Stability Avoidance Modularization
non-intrusive intrusive
few many
intrusiveness
earlier recent
recency as a solution at Netflix
scope of problems solved
Modularization
Just-in-Time Shading
REVERSE PROXY
Revisiting Conflict Resolution
app
lib1 lib2
guava:16
guava:14
guava:14Unshaded
Shaded
lib3
guava:14
Public API Methodsapp
A
guava:16
guava:14 Unshaded
Shaded
public Multiset<String> wordCount(String word) public int totalSales(RangeSet<Integer> hourRange)
Hard if A has API methods such as
Thanks!(We are hiring)