Post on 25-Jun-2015
description
transcript
Distributed DevelopmentProject FreeSpaceDmitri Nesterukdmitri@activemesa.com
Alexey Suvorovalexey@activemesa.com
This Talk Is About…
SpeedHow to get things done faster
QualityHow to get feedback fasterHow to get more testing done
ManageabilityCloud monitoring & controlDecentralization/fault-tolerance
Not only development!
In large/complex projects
IDE interaction is slowCode analysis is slowCompilation is slowTesting is slow(Re)deployment is slow
IDE interaction is slow
IDEs are slow, but we cannot ditch themWe have nearly few software options for optimizing IDEs
E.g., VS is both disk I/O-bound (SSD a must) and CPU-boundWe cannot relocate, e.g., the ReSharper cache into a distributed service
IDEs can be spawned on same project in many machines
Multiple screens/remote desktop windowsSynchronizable with Dropbox, SugarSync, etc.But project/solution reloads in VS will kill you
Code analysis is slow
In-depth analysis of either compiled or source code is computationally intensiveNDepend, FxCop and others can all be run remotely
Not just on the build serverMost of these tools output report files
Can send these to originSome of these tools can be made to work on per-file/per-project rather than per-solution
Compilation is slow
Compilation isC#/VB.NET – acceptableF# – slowC++ – atrociously slow
Made worse by pre/post-build
PostSharpEntity FrameworkCode ContractsMolesEtc.
VS compilation process inefficient
Will rebuild projects that haven’t changedWill not parallelize by default
MSBuild is parallelizable
/m:nCan spawn multiple processes
Testing is (painfully) slow
Unit testing is badly parallelizedMbUnit’s [Parallelizable]Same in NUnit 3
Can easily parallelize at different granularity
Test case/methodTest fixtureTest assembly
Fear of Builds/Tests
Developers loath to compile or run tests too frequentlyDisruption of focus leads them to
Surf the webGo for coffee<insert your pastime here>
Everyone losesLoss of concentration/motivationDevelopers never ‘in the zone’TDD does not work
Who cares?
DevelopersFixed salaryDon’t care about TTMAccustomed to substandard tools/equipmentView compilation as a one-time processDon’t care about frequent/continuous testingSee manual deployment as normal
EmployersMore concerned with saving money than getting things doneUninformed about good/best practicesNot concerned with quick delivery (in case of service companies that charge by the hour)
Problem: nobody recognizes compilation/testing as wasted
time
How to speed things up?
Optimize or buy a faster computerSSDsMore RAMFaster CPUsCostly! Has to be done for every developer.
Alternatively… use existing infrastructure
Both physical (e.g., dev machines) and virtualDistribute workload between machinesUse idle resources – no need to buy new machines.
Status Quo
Computers get fasterMore cores per CPUFaster hard drives (SSD, hybrid)
Software gets more demandingWindows eats more and more RAM & HDDVS is slowerEveryone else follows suit
The overall development experience isnot getting any better
Why Distributed?
Resource under-utilizationA typical enterprise (IT-specific or not) is unlikely to use 100% of processing resources
Resource overloadBottlenecks in servers
Resource costsServer-grade hardwareReliability concerns (e.g. RAID)
Three Pillars of Distribution
Get data on everyone’s machineCloud storage/file syncVerification necessary
Get machines chatting with one another
XMPP client on each nodeLoad balance and optimize execution planSend commands to do work, get results
Work items synchronized via cloud storage
Redundancy/reliability guaranteesIntegrate with existing systems
Easy because XMPP uses XML
Scale Vectors
CoreBetter support for Multi-CoreExists in some cases
MSBuildMbUnit (+ NUnit 3)
Could be leveraged in the general case
Not easy!Needs to mind end user’s preferences
MachineSupport arbitrary networks (both on- and off-site)Need to control code sync (security)Can go for full resource utilization (esp. off-hours)
Speculative processingE.g., Monte-Carlo simulations
Operations which are prohibitively resource-intensive
E.g., mutation tests
Leveraging the Model
CompilationCompiling on dev’s machine is counterproductiveCompilation of some languages (C++, Scala) takes far too much time
But the problem exists everywhere (.Net, Java)
Deployment
TestingLarge test base cannot be run on a dev’s machineCI is not the answer
Constrained to a single machineCan be distributed, but not straightforward
Code analysisVery costly
Coverage analysisExtremely costly
Compilation
MSBuildBuilds all major types of VS projectsCan parallelize locally (/m:n, n=# of processes)
Builds block VSBuild on the UI thread
Builds often inefficientCannot build only projects affected by changes
Cannot use multiple machines
Distributed Compilation
Dramatically speed up solution buildDetermine project dependenciesBuild different projects on separate machinesUse multiple MSBuild processes per machine
Depending on CPU count & power
Does not distract the developerDevelopment machine usable without interruptionQuicker feedback on errorsAllows to instrument a continuous build policy
Build on ever file save
Distributed Testing
Testing is slowUnit testing is largely not parallelized
MbUnit [Paralellizable]Nunit will only support it in v.3
Not parallelized between several machines
Testing in specific environment difficultRequires complicated (possibly manual) deployment processes
Developer typically only tests on their own box (+ maybe CI server)
Distributed testing ensures tests work everywhere
Side Effects
Side effects are unwelcome on users’ machinesEnvironment changes may have undesired consequencesBuilds are typically exempt from this
They do not affect anything beyond solution work folders
Unit tests may or may not affect host systemIntegration tests typically do affect hosts
Require ‘clean’ set-ups
Isolation
No side-effectsIrrelevant, just take care of load
Side effectsProcess-level virtualization (for existing machines)
JauntePEApp-V
Virtual machinesHyper-VESX
Virtualized Testing
Creation of multiple physical nodes is costlyPhysical machine re-configuration takes too much timeCan configure a virtual test environment with
Hyper-VSystem Center Virtual Machine Manager
Virtual/physical migrations
Different hardware requirementsMulti-CPU systemFast disksVery large amounts of RAM
Project FreeSpace
Private Cloud InfrastructureXMPP + file sync
Single-MSI deploymentPlugin architectureFully self-updatingDecentralized service orchestration
Self-organizingEach node has identical capability
Easy to administer
Project FreeSpace Features
Distributed CompilationInitially MSBuild
Distributed Unit TestingInitially via Gallio test automation framework
Distributed Integration TestingVirtual machine managementInitially via Hyper-V
Distributed deploymentE.g., create new VM for testers with appropriate binaries etc.
Benefit Summary
Better than Continuous IntegrationBetter than Continuous TestingBetter than local compilationBetter than local testing
Better Than Continuous Integration
Good for long-running builds/testsHappens on a single machine
Can set up, e.g., multiple instances, but it’s not straightforwardNot designed for distributed builds
Not optimized for idle processingAssumes server is dedicated
Does not give immediate feedbackTypically works on commit
I.e., detects source control changes
Better Than Continuous Testing
Testing is often more costly than compilationTypically, tests run on commitContinuous testing (e.g., Mighty Moose) systems ensure that
Tests run on each saveOnly tests affected by changes are executedFast feedback…
But not instant – you still need to recompile.
Given the option, why not build/test thingsall the time?
Better Than Local Compilation
Does not block IDEScales across your networkMuch faster buildsImmediate feedback
Better Than Local Testing
Much faster recompilationTests do not tax developer CPUAllows for immediate testing in different environmentsTests happen in parallel (where possible)
That’s all!
Questions?