Transparent Process Migration: Transparent Process Migration: Design Alternatives and the Design Alternatives and the

SpriteSprite Implementation Implementation

Fred Douglis and John Fred Douglis and John OusterhoutOusterhout

PurposesPurposes

Utilize idle workstationsUtilize idle workstations Reclaim ownership of workstations Reclaim ownership of workstations

when no longer idlewhen no longer idle

MainMain GoalsGoals

TransparentTransparent Users own their workstationsUsers own their workstations High performanceHigh performance Low ComplexityLow Complexity Easy to useEasy to use

AlternativesAlternatives rshrsh Nichol’s Butler SystemNichol’s Butler System AccentAccent CharlotteCharlotte VV LOCUSLOCUS 7up7up ““A Brief Survey of Systems Providing A Brief Survey of Systems Providing

Process or Object Migration Facilities”Process or Object Migration Facilities”

System FeaturesSystem Features Idle hosts are plentifulIdle hosts are plentiful Users own their Users own their

workstationsworkstations Most programs are Most programs are

short livedshort lived Sprite uses kernel Sprite uses kernel

calls (as opposed to calls (as opposed to message passing)message passing)

Sprite provides Sprite provides networking networking mechanisms that mechanisms that allow for ease of allow for ease of implementationimplementation

PoliciesPolicies

Which processes do we Which processes do we migratemigrate

Migrate only at initiation or Migrate only at initiation or after running a whileafter running a while

How do we select idle How do we select idle machinesmachines

Who makes the decisionsWho makes the decisions

Possible OptionsPossible Options

Pool of processorsPool of processors No policy support (rsh)No policy support (rsh) Automated selection of idle hosts but Automated selection of idle hosts but

other decisions are only partially other decisions are only partially automatedautomated

Selection of Idle HostsSelection of Idle Hosts

Each machine has a load-average Each machine has a load-average daemon which monitors usagedaemon which monitors usage

Load-average daemon notifies Load-average daemon notifies central migration server when idlecentral migration server when idle

Request for idle host is sent to server Request for idle host is sent to server processprocess

What determines if a host is What determines if a host is idle?idle?

No keyboard or mouse movement for No keyboard or mouse movement for a certain perioda certain period

Fewer runnable processes than Fewer runnable processes than processorsprocessors

Which idle host do we use?Which idle host do we use?

The host that is idle for the longest The host that is idle for the longest period of timeperiod of time

The host with the most amount of The host with the most amount of memorymemory

Transparency DefinitionTransparency Definition

Behavior unaffected by migrationBehavior unaffected by migration Appearance to rest of world Appearance to rest of world

unaffected by migrationunaffected by migration

Achieving TransparencyAchieving Transparency

Changed file system so every Changed file system so every machine sees same name spacemachine sees same name space

Transfer state information from Transfer state information from source to targetsource to target

Forward kernel calls homeForward kernel calls home Ad hoc techniques for specific kernel Ad hoc techniques for specific kernel

calls such as forkcalls such as fork

State InformationState Information

Virtual MemoryVirtual Memory Open FilesOpen Files Message ChannelsMessage Channels Execution StateExecution State Other (pid, user id, working directory, Other (pid, user id, working directory,

signal masks and handlers, resource signal masks and handlers, resource usage stats, references to usage stats, references to parent/child, etc.)parent/child, etc.)

Possible Solutions for Handling Possible Solutions for Handling StateState

Transfer StateTransfer State Arrange for forwardingArrange for forwarding Ignore state and sacrifice Ignore state and sacrifice

transparencytransparency Disallow migration for Disallow migration for

certain processescertain processes

Virtual Memory MechanismVirtual Memory Mechanism

Transfer at migration (Charlotte & Transfer at migration (Charlotte & LOCUS)LOCUS)

Precopying (V)Precopying (V) Lazy copy (Accent)Lazy copy (Accent) Lazy copy … sorta (Sprite)Lazy copy … sorta (Sprite)

Open File MechanismOpen File Mechanism

Close file on source reopen on targetClose file on source reopen on target Notify file is in use on target then Notify file is in use on target then

notify no longer in use on sourcenotify no longer in use on source Special server code for migrating Special server code for migrating

files – Source contacts target, target files – Source contacts target, target contacts central server, server contacts central server, server contacts hostcontacts host

PCB and Other State PCB and Other State InformationInformation

Home and target need PCB for a Home and target need PCB for a processprocess

PCB on home contains only enough PCB on home contains only enough fields to locate processfields to locate process

PCB on target is completePCB on target is complete Other state is either part of PCB or Other state is either part of PCB or

can be transferred directlycan be transferred directly

EvictionEviction

Load-average daemon checks for Load-average daemon checks for foreign processes and evicts themforeign processes and evicts them

Process migrated back to hostProcess migrated back to host Can also occur if centralized server Can also occur if centralized server

receives request for idle host when receives request for idle host when none are availablenone are available

Residual DependenciesResidual Dependencies

Definition: Ongoing need for home to Definition: Ongoing need for home to maintain data structures or provide maintain data structures or provide functionality after migrationfunctionality after migration

Problems with Residual Problems with Residual DependenciesDependencies

Decrease in reliabilityDecrease in reliability Decrease in performanceDecrease in performance Increase in complexityIncrease in complexity

Other Problems: Other Problems: Interdependence of Migration Interdependence of Migration

and Kerneland Kernel

Migration involves state manipulated by Migration involves state manipulated by every major module in kernelevery major module in kernel

Changes in one affects the otherChanges in one affects the other SolutionsSolutions

• Migration version numbers: Disallow Migration version numbers: Disallow migration between processes with migration between processes with different version numbersdifferent version numbers

• Avoid centralizing migration code Avoid centralizing migration code

AlgorithmAlgorithm Process initiates communication with central Process initiates communication with central

server (opens a special file, OS passes the open server (opens a special file, OS passes the open operation onto server)operation onto server)

Process requests idle hostsProcess requests idle hosts Centralized server selects idle host and sends Centralized server selects idle host and sends

information to the home machineinformation to the home machine Process is signaled to make it trap into kernelProcess is signaled to make it trap into kernel Source contacts target to confirm it is running, Source contacts target to confirm it is running,

available for migration and has same version available for migration and has same version numbernumber

Algorithm ContinuedAlgorithm Continued Kernel on target allocates new PCB and returns a Kernel on target allocates new PCB and returns a

token identifier token identifier Pre-migration routine obtains size of Pre-migration routine obtains size of

encapsulated data and takes necessary actionsencapsulated data and takes necessary actions Source kernel allocates buffer to hold Source kernel allocates buffer to hold

encapsulated stateencapsulated state Call encapsulation routine to place encapsulated Call encapsulation routine to place encapsulated

information into portion of bufferinformation into portion of buffer Source kernel passes encapsulated state to target Source kernel passes encapsulated state to target

kernel via RPCkernel via RPC

Algorithm ContinuedAlgorithm Continued Target begins de-encapsulation routineTarget begins de-encapsulation routine Some processes also require a post-Some processes also require a post-

migration routine to clean up remaining migration routine to clean up remaining statestate

Source kernel frees buffer and informs Source kernel frees buffer and informs target to resume process executiontarget to resume process execution

After process completes return host to After process completes return host to pool of idle hostspool of idle hosts

Close communication channel to central Close communication channel to central serverserver

Practical ApplicationsPractical Applications

pmake: used to parallelize make pmake: used to parallelize make command by invoking as many command by invoking as many commands in parallel as there are commands in parallel as there are idle hostsidle hosts

mig: takes as argument a shell mig: takes as argument a shell command and executes it on an idle command and executes it on an idle machinemachine

Performance: Migration Performance: Migration OverheadOverhead

ActionAction Time/RateTime/Rate

Select and release idle hostSelect and release idle host 36 ms36 ms

Migrate null processMigrate null process 76 ms76 ms

Transfer info for open filesTransfer info for open files 9.4 ms/file9.4 ms/file

Flush modified file blocksFlush modified file blocks 480 Kbyte/s480 Kbyte/s

Flush modified pagesFlush modified pages 660 Kbyte/s660 Kbyte/s

Transfer exec argumentsTransfer exec arguments 480 Kbyte/s480 Kbyte/s

Fork, exec null process Fork, exec null process w/migration, wait for child to w/migration, wait for child to exitexit

81 ms81 ms

Fork, exec null process locally, Fork, exec null process locally, wait for child to exitwait for child to exit

46 ms46 ms

Performance: Application Performance: Application PerformancePerformance

0

20

40

60

80

100

120

140

160

180

pmake LaTeX rcp fork gettime

local

remote

Performance: Usage PatternsPerformance: Usage Patterns

Remote processes were 31% of Remote processes were 31% of executionexecution

Evictions have been rareEvictions have been rare

ConclusionsConclusions Overall improvement from using idle hosts Overall improvement from using idle hosts

can be substantial.can be substantial. Remote execution accounts for a sizeable Remote execution accounts for a sizeable

portion of execution in Sprite. There are portion of execution in Sprite. There are still a number of idle hosts.still a number of idle hosts.

Execution time for a process is higher if Execution time for a process is higher if executed remotely but not noticeable by executed remotely but not noticeable by human standards.human standards.

Cost of transferring address space and Cost of transferring address space and flushing modified file blocks dominates flushing modified file blocks dominates migrating long running processesmigrating long running processes

Future WorkFuture Work More applications that More applications that

make use of migrationmake use of migration Remote execution on Remote execution on

machines of different machines of different typestypes

Mechanism for Mechanism for migration of processes migration of processes that share memorythat share memory

Detect when users Detect when users and not just and not just workstations are idleworkstations are idle

Questions???Questions???

If you dare…If you dare…