Date post: | 08-May-2015 |
Category: |
Technology |
Upload: | cachet-software-solutions-ltd |
View: | 383 times |
Download: | 0 times |
©2013 CloudBees, Inc. All Rights Reserved 1©2013 CloudBees, Inc. All Rights Reserved
Next Step in Automation:Elastic Build EnvironmentKohsuke Kawaguchi / CloudBees, [email protected] / @kohsukekawa
©2013 CloudBees, Inc. All Rights Reserved 2
Have You Met Jenkins? http://jenkins-ci.org/
©2013 CloudBees, Inc. All Rights Reserved 3
32%
©2013 CloudBees, Inc. All Rights Reserved 4
18%
©2013 CloudBees, Inc. All Rights Reserved 5
©2013 CloudBees, Inc. All Rights Reserved 6
My Jenkins around 2006
©2013 CloudBees, Inc. All Rights Reserved 7
©2013 CloudBees, Inc. All Rights Reserved 8
©2013 CloudBees, Inc. All Rights Reserved 9
Workload
http://www.flickr.com/photos/gbyrnes/912576883/
©2013 CloudBees, Inc. All Rights Reserved 10
If only we had more computers…
• Just building & testing them all…• Running tests more frequently• Testing individual commits
©2013 CloudBees, Inc. All Rights Reserved 11
What you don’t wanthttp://www.flickr.com/photos/drocpsu/8546730021/
©2013 CloudBees, Inc. All Rights Reserved 12
Just enough computersjust in time
Elasticity
©2013 CloudBees, Inc. All Rights Reserved 13
My Jenkins around 2007
©2013 CloudBees, Inc. All Rights Reserved 14
©2013 CloudBees, Inc. All Rights Reserved 15
©2013 CloudBees, Inc. All Rights Reserved 16
©2013 CloudBees, Inc. All Rights Reserved 17http://www.flickr.com/photos/drocpsu/8546730021/
©2013 CloudBees, Inc. All Rights Reserved 18
18
But not this, either
©2013 CloudBees, Inc. All Rights Reserved 19
Just enough computersof the right kind
just in time
Elasticity!
©2013 CloudBees, Inc. All Rights Reserved 20http://www.flickr.com/photos/82219206@N00/7003641975/
©2013 CloudBees, Inc. All Rights Reserved 21
Correct answer
• Test assumes a fixture running on port 8080– Doesn’t check if it’s already being used
• If another test runs at the same time…?
©2013 CloudBees, Inc. All Rights Reserved 22http://www.flickr.com/photos/82219206@N00/7003641975/
©2013 CloudBees, Inc. All Rights Reserved 23
Correct answer
• Because of “pkill -f -9 tomcat” cleanup
©2013 CloudBees, Inc. All Rights Reserved 24
Isolation
http://www.flickr.com/photos/jumilla/8667648797/
©2013 CloudBees, Inc. All Rights Reserved 25
Isolation
• At odds with large multi-core systems
• x86 virtual machines• User isolation• Kernel containers
©2013 CloudBees, Inc. All Rights Reserved 26
!
©2013 CloudBees, Inc. All Rights Reserved 27http://www.flickr.com/photos/82219206@N00/7003641975/
©2013 CloudBees, Inc. All Rights Reserved 28
Correct answer
• Same Maven ID, two different jars
• Different projects designate different ones• Local cache gets cleaned up periodically
• Whichever first runs after cache cleanup “wins”
©2013 CloudBees, Inc. All Rights Reserved 29http://www.flickr.com/photos/82219206@N00/7003641975/
©2013 CloudBees, Inc. All Rights Reserved 30
Correct answer
• Test script leaves background daemon process behind
• Over time it’ll slowly choke slaves
©2013 CloudBees, Inc. All Rights Reserved 31
©2013 CloudBees, Inc. All Rights Reserved 32
Throw away & create new
Elasticity!
©2013 CloudBees, Inc. All Rights Reserved 33
©2013 CloudBees, Inc. All Rights Reserved 34
Ladder to Cloud
Single
MultipleElastic
©2013 CloudBees, Inc. All Rights Reserved 35
Parallel Testing
Test Group #1 Test Group #2 Test Group #3
©2013 CloudBees, Inc. All Rights Reserved 36
Parallel Testing
foo #10 Test Group #1
foo #11 Test Group #2
foo #12 Test Goup #3
©2013 CloudBees, Inc. All Rights Reserved 37
Validated Merge: Jenkins Enterprise by CloudBees
upstreamrepo
gate repo
©2013 CloudBees, Inc. All Rights Reserved 38
Recap: Elasticity Benefits
• Just-in-time capacity• Diversity without overhead• Isolation• Productivity gain– parallel testing– validated merge– keep failed tests on the side
©2013 CloudBees, Inc. All Rights Reserved 39
Solid OSS Elasticity Plugins
• EC2 plugin• Jclouds plugin– OpenStack, CloudStack
• Launch and tear down slaves on demand
©2013 CloudBees, Inc. All Rights Reserved 40Image © http://sfcitizen.com/blog/wp-content/uploads/2011/11/6302790910_c4eb865892_o-copy.jpg
©2013 CloudBees, Inc. All Rights Reserved
Demo
©2013 CloudBees, Inc. All Rights Reserved 41
• Master– Serves HTTP requests– Stores all important info
• Slaves– 280KB single jar– Assumed to be
unreliable– Scale to 100s
Distributed builds with Jenkins
Master
slave1
slave2
slave3
slave4
slave5
slave6
slave7
slave8
©2013 CloudBees, Inc. All Rights Reserved 42
Lots More Way To Deploy Slaves
• Good old shell scripting
$ java -jar slave.jar -jnlpUrl URL
©2013 CloudBees, Inc. All Rights Reserved 43
How Flexible Does It Get?
• Slave agent is a little Java program– Runs locally on the machine that it controls– Access files, fork processes, etc., on behalf
of master
• Communication with master– Just needs a bi-di byte stream
• No shared file system, no network topology constraints, etc
©2013 CloudBees, Inc. All Rights Reserved 44
Static Virtual Machines As Build Slaves
• Allocate lots of identical VMs– Better consistency, lower overhead
• Curve up 1 big machine to small VMs• Throw away > maintain
©2013 CloudBees, Inc. All Rights Reserved 45
Labels
• Refer to slavesbut not by name
Name: alphaLabel: windows 32bit
Name: bravoLabel: linux 32bit
Name: charlieLabel: linux 64bit
©2013 CloudBees, Inc. All Rights Reserved 46
Labels
Name: alphaLabel: windows 32bit
Name: bravoLabel: linux 32bit
Name: charlieLabel: linux 64bit
This job needs to run on “linux”
©2013 CloudBees, Inc. All Rights Reserved 47
Labels
Name: alphaLabel: windows 32bit
Name: bravoLabel: linux 32bit
Name: charlieLabel: linux 64bit
This job needs to run on “linux && 32bit”
©2013 CloudBees, Inc. All Rights Reserved 48
More Label Benefits
• Labels can be anything– Geographic locations– Availability of proprietary software
installed
• Better resource utilization– Jenkins have maximum freedom
• Resilient to node addition/removal– Treat servers like livestocks, not like pets
©2013 CloudBees, Inc. All Rights Reserved 49
Load Statistics Monitoring
49
©2013 CloudBees, Inc. All Rights Reserved 50
“Cloud” plugin
©2013 CloudBees, Inc. All Rights Reserved 51
EC2 Plugin
• Label → Amazon Machine Image (AMI)• Integrates with various AWS features– Spot Instance– Identity and Access Management (IAM)– Virtual Private Cloud (VPC)
©2013 CloudBees, Inc. All Rights Reserved 52
jclouds plugin
• CloudStack, OpenStack, and many more
• “One-time use” support
Jenkins jclouds cloud
©2013 CloudBees, Inc. All Rights Reserved 53
Then Adjust Capacity based on it
©2013 CloudBees, Inc. All Rights Reserved 54
Jenkins Enterprise by CloudBees
• VMWare auto-scaling plugin– Snapshot– Power on-off management– Hypervisor-aware scheduling– Folder based pooling– VMWare tools integration– One-time use support
©2013 CloudBees, Inc. All Rights Reserved 55
Mansion
Slave Slave
Slave Slave
CloudBees DEV@cloud
©2013 CloudBees, Inc. All Rights Reserved 56
Maven
Git
Ant
Mercurial
Gradle
Subversion
Linux Kernel
Hardware
Linux Container = zero cost virtualization
©2013 CloudBees, Inc. All Rights Reserved 57
For OS X
Maven
Git
XCode
Git
XCode
Subversion
OS X OS X OS X
QEMU QEMU QEMU
Linux Kernel
Apple Hardware
©2013 CloudBees, Inc. All Rights Reserved 58
Kernel Same-page Merging
OS X OS X OS X
©2013 CloudBees, Inc. All Rights Reserved 59
©2013 CloudBees, Inc. All Rights Reserved 60
Mansion
Slave
Slave
Slave
Workspace 1
Workspace 2
Workspace 3
Workspace 4
©2013 CloudBees, Inc. All Rights Reserved 61
Slave
Slave
Workspace
Workspace’’
Workspace’
©2013 CloudBees, Inc. All Rights Reserved 62
Slave
Slave
~/.m2/repository
~/.m2/repository
~/.m2/repository
©2013 CloudBees, Inc. All Rights Reserved 63
この苦しみから解脱する方法はないかと
• お釈迦様の絵
• 徐々に蓄積してきたノウハウ・ツールを大公開
Elasticity Nirvana?
©2013 CloudBees, Inc. All Rights Reserved 64
©2013 CloudBees, Inc. All Rights Reserved 65
Testing things in piecesTraditional Approach
©2013 CloudBees, Inc. All Rights Reserved 66
Testing the whole is hard
• Database• Load balancer• Messaging layer• Identity service• …
©2013 CloudBees, Inc. All Rights Reserved 67http://www.flickr.com/photos/46471983@N00/339605220/
Substitute vs Mock
©2013 CloudBees, Inc. All Rights Reserved 68
What you really want = real deal!
Test and production environment should be identical
… which means automating:– Database– Load balancer–Messaging layer– Identity service–…
©2013 CloudBees, Inc. All Rights Reserved 69
The Challenge
Infrastructure as Code
©2013 CloudBees, Inc. All Rights Reserved 70
The Real Challenge
Elasticity
©2013 CloudBees, Inc. All Rights Reserved 71
Single
MultipleElastic
The same evolution ladder
aka PaaS
©2013 CloudBees, Inc. All Rights Reserved 72