Date post: | 11-Mar-2018 |
Category: |
Documents |
Upload: | duonghuong |
View: | 220 times |
Download: | 3 times |
1
CSE 513Introduction to Operating Systems
Class 9 - Distributed andMultiprocessor Operating Systems
Jonathan WalpoleDept. of Comp. Sci. and Eng.
Oregon Health and Science University
2
Why use parallel or distributed systems?
q Speed - reduce time to answerq Scale - increase size of problemq Reliability - increase resilience to errorsq Communication - span geographical distance
Multiprocessor, multi-computer anddistributed architectures
v shared memory multiprocessorv message passing multi-computer (cluster)v wide area distributed system
6
Multiprocessor systems
q Definition:v A computer system in which two or more CPUs
share full access to a common RAMq Hardware implements shared memory among
CPUsq Architecture determines whether access times
to different memory regions are the samev UMA - uniform memory accessv NUMA - non-uniform memory access
11
NUMA multiprocessors
q Single address space visible to all CPUsq Access to remote memory via commands
- LOAD- STORE
q Access to remote memory slower than to localmemory
q Compilers and OS need to be careful aboutdata placement
12
Directory-based NUMA multiprocessors
(a) 256-node directory based multiprocessor(b) Fields of 32-bit memory address(c) Directory at node 36
13
Operating systems for multiprocessors
q OS structuring approachesv Private OS per CPUv Master-slave architecturev Symmetric multiprocessing architecture
q New problemsv multiprocessor synchronizationv multiprocessor scheduling
14
The private OS approach
q Implications of private OS approachv shared I/O devicesv static memory allocationv no data sharingv no parallel applications
15
The master-slave approach
q OS only runs on master CPUv Single kernel lock protects OS data structuresv Slaves trap system calls and place process on scheduling
queue for masterq Parallel applications supported
v Memory shared among all CPUsq Single CPU for all OS calls becomes a bottleneck
16
Symmetric multiprocessing (SMP)
q OS runs on all CPUsv Multiple CPUs can be executing the OS simultaneouslyv Access to OS data structures requires synchronizationv Fine grain critical sections lead to more locks and more
parallelism … and more potential for deadlock
17
Multiprocessor synchronization
q Why is it different compared to singleprocessor synchronization?v Disabling interrupts does not prevent memory
accesses since it only affects “this” CPUv Multiple copies of the same data exist in caches of
different CPUs• atomic lock instructions do CPU-CPU communication
v Spinning to wait for a lock is not always a bad idea
20
Spinning versus switching
q In some cases CPU “must” waitv scheduling critical section may be held
q In other cases spinning may be more efficientthan blockingv spinning wastes CPU cyclesv switching uses up CPU cycles alsov if critical sections are short spinning may be better
than blockingv static analysis of critical section duration can
determine whether to spin or blockv dynamic analysis can improve performance
21
Multiprocessor scheduling
q Two dimensional scheduling decisionv time (which process to run next)v space (which processor to run it on)
q Time sharing approachv single scheduling queue shared across all CPUs
q Space sharing approachv partition machine into sub-clusters
22
Time sharing
q Single data structure used for schedulingq Problem - scheduling frequency influences
inter-thread communication time
23
Interplay between scheduling and IPC
q Problem with communication between two threadsv both belong to process Av both running out of phase
24
Space sharing
q Groups of cooperating threads can communicate atthe same timev fast inter-thread communication time
25
Gang scheduling
q Problem with pure space sharingv Some partitions are idle while others are overloaded
q Can we combine time sharing and space sharingand avoid introducing scheduling delay into IPC?
q Solution: Gang Schedulingv Groups of related threads scheduled as a unit (gang)v All members of gang run simultaneously on different
timeshared CPUsv All gang members start and end time slices together
28
Multi-computers
q Also known asv cluster computersv clusters of workstations (COWs)
q Definition:Tightly-coupled CPUs that do notshare memory
29
Multi-computer interconnection topologies
(a) single switch(b) ring(c) grid
(d) double torus(e) cube(f) hypercube
31
Network interfaces in a multi-computer
q Network co-processors may off-loadcommunication processing from the main CPU
32
OS issues for multi-computers
q Message passing performance
q Programming modelv synchronous vs asynchornous message passingv distributed virtual memory
q Load balancing and coordinated scheduling
33
Optimizing message passing performance
q Parallel application performance is dominated bycommunication costsv interrupt handling, context switching, message
copying …
q Solution - get the OS out of the loopv map interface board to all processes that need itv active messages - give interrupt handler address of
user-bufferv sacrifice protection for performance?
34
CPU / network card coordination
q How to maximize independence between CPU andnetwork card while sending/receiving messages?v Use send & receive rings and bit-mapsv one always sets bits, one always clears bits
35
Blocking vs non-blocking send calls
q Minimum services providedv send and receive
commands
q These can be blocking(synchronous) or non-blocking (asynchronous)calls
(a) Blocking send call
(b) Non-blocking send call
36
Blocking vs non-blocking calls
q Advantages of non-blocking callsv ability to overlap computation and communication
improves performance
q Advantages of blocking callsv simpler programming model
37
Remote procedure call (RPC)
q Goalv support execution of remote proceduresv make remote procedure execution indistinguishable
from local procedure executionv allow distributed programming without changing the
programming model
38
Remote procedure call (RPC)
q Steps in making a remote procedure callv client and server stubs are proxies
39
RPC implementation issues
q Cannot pass pointersv call by reference becomes copy-restore (at best)
q Weakly typed languagesv Client stub cannot determine size of reference
parametersv Not always possible to determine parameter types
q Cannot use global variablesv may get moved (replicated) to remote machine
q Basic problem - local procedure call relies onshared memory
40
Distributed shared memory (DSM)
q Goalv use software to create the illusion of shared
memory on top of message passing hardwarev leverage virtual memory hardware to page fault on
non-resident pagesv service page faults from remote memories instead
of from local disk
42
Page replication in DSM systems
Replication
(a) Pages distributed on 4machines
(b) CPU 0 reads page 10
(c) CPU 1 reads page 10
44
Strong memory consistency
P1
P2
P3
P4
W1
W2
W3
W4
R2
R1
q Total order enforces sequential consistencyv intuitively simple for programmers, but very costly to
implement
v not even implemented in non-distributed machines!
45
Scheduling in multi-computer systems
q Each computer has its own OSv local scheduling applies
q Which computer should we allocate a task toinitially?v Decision can be based on load (load balancing)v load balancing can be static or dynamic
46
Graph-theoretic load balancing approach
Process
q Two ways of allocating 9 processes to 3 nodesq Total network traffic is sum of arcs cut by node
boundariesq The second partitioning is better
47
Sender-initiated load balancing
q Overloaded nodes (senders) off-load work to underloadednodes (receivers)
48
Receiver-initiated load balancing
q Underloaded nodes (receivers) request work from overloadednodes (senders)
50
Distributed systems
q Definition: Loosely-coupled CPUs that do notshare memoryv where is the boundary between tightly-coupled and
loosely-coupled systems?
q Other differencesv single vs multiple administrative domainsv geographic distributionv homogeneity vs heterogeneity of hardware and
software
54
OS issues for distributed systems
q Common interfaces above heterogeneoussystemsv Communication protocolsv Distributed system middleware
q Choosing suitable abstractions for distributedsystem interfacesv distributed document-based systemsv distributed file systemsv distributed object systems
58
Distributed system middleware models
q Document-based systemsq File-based systemsq Object-based systems
60
Document-based middleware
How the browser gets a pageq Asks DNS for IP addressq DNS replies with IP addressq Browser makes connectionq Sends request for specified pageq Server sends fileq TCP connection releasedq Browser displays textq Browser fetches, displays images
61
File-based middleware
q Design issuesv Naming and name resolutionv Architecture and interfacesv Caching strategies and cache consistencyv File sharing semanticsv Disconnected operation and fault tolerance
62
Naming
(b) Clients with the same view of name space(c) Clients with different views of name space
63
Naming and transparency issues
q Can clients distinguish between local and remote files?
q Location transparencyv file name does not reveal the file's physical storage
location.
q Location independencev the file name does not need to be changed when the
file's physical storage location changes.
64
Global vs local name spaces
q Global name spacev file names are globally uniquev any file can be named from any node
q Local name spacesv remote files must be inserted in the local name spacev file names are only meaningful within the calling nodev but how do you refer to remote files in order to insert
them?• globally unique file handles can be used to map remote
files to local names
65
Building a name space with super-root
q Super-root / machine name approachv concatenate the host name to the names of files stored on
that hostv system-wide uniqueness guaranteedv simple to located a filev not location transparent or location independent
66
Building a name space using mountingq Mounting remote file systems
v exported remote directory is imported and mounted ontolocal directory
v accesses require a globally unique file handle for the remotedirectory
v once mounted, file names are location-transparent• location can be captured via naming conventions
v are they location independent?• location of file vs location of client?• files have different names from different places
69
NSF name space
q Server exports a directory
q mountd: provides a unique file handle for the exporteddirectory
q Client uses RPC to issue nfs_mount request to server
q mountd receives the request and checks whetherv the pathname is a directory?v the directory is exported to this client?
70
NFS file handles
q V-node containsv reference to a file handle for mounted remote filesv reference to an i-node for local files
q File handle uniquely names a remote directoryv file system identifier: unique number for each file system (in UNIX
super block)v i-node and i-node generation number
v-nodei-nodeFile handle
File System identifier i-nodei-node generation
number
71
Mounting on-demand
q Need to decide where and when to mount remotedirectories
q Where? - Can be based on conventions to standardizelocal name spaces (ie., /home/username for user homedirectories)
q When? - boot time, login time, access time, …?q What to mount when?
v How long does it take to mount everything?v Do we know what everything is?v Can we do mounting on-demand?
q An automounter is a client-side process that handles on-demand mountingv it intercepts requests and acts like a local NFS server
72
Distributed file system architectures
q Server sidev how do servers export filesv how do servers handle requests from clients?
q Client sidev how do applications access a remote file in the same way
as a local file?
q Communication layerv how do clients and servers communicate?
73
Local access architectures
q Local access approachv move file to clientv local access on clientv return file to serverv data shipping
approach
74
Remote access architectures
q Remote accessv leave file on serverv send read/write operations
to serverv return results to clientv function shipping approach
75
File-level interface
q Accesses can be supported at either the filegranularity or block granularity
q File-level client-server interfacev local access model with whole file movement and
cachingv remote access model client-server interface at
system call levelv client performs remote open, read, write, close calls
76
Block-level interface
q Block-level client-server interfacev client-server interface at file system or disk block
levelv server offers virtual disk interfacev client file accesses generate block access requests
to serverv block-level caching of parts of files on client
78
NFS server side
q Mountdv server exports directory via mountdv mountd provides the initial file handle for the exported
directoryv client issues nfs_mount request via RPC to mountdv mountd checks if the pathname is a directory and if the
directory is exported to the client
q nfsd: services NFS RPC calls, gets the data from itslocal file system, and replies to the RPCv Usually listening at port 2049
q Both mountd and nfsd use RPC
79
Communication layer: NFS RPC Calls
q NFS / RPC uses XDR and TCP/IPq fhandle: 64-byte opaque data (in NFS v3)
v what’s in the file handle?
status, fattrfhandle, offset, count, datawrite
status, fhandle, fattrdirfh, name, fattrcreate
status, fattr, datafhandle, offset, countread
status, fhandle, fattrdirfh, namelookup
ResultsInput argsProc.
80
NFS file handles
q V-node containsv reference to a file handle for mounted remote filesv reference to an i-node for local files
q File handle uniquely names a remote directoryv file system identifier: unique number for each file system (in UNIX
super block)v i-node and i-node generation number
v-nodei-nodeFile handle
File System identifier i-nodei-node generation
number
81
NFS client side
q Accessing remote files in the same way asaccessing local files requires kernel supportv Vnode interface
read(fd,..) struct file
ModeVnodeoffset
V_data
fs_op
struct vnode
{int (*open)(); int (*close)(); int (*read)(); int (*write)(); int (*lookup)(); … }
processfile table
82
Caching vs pure remote service
• Network traffic?– caching reduces remote accesses ⇒ reduces network traffic– caching generates fewer, larger, data transfers
• Server load?– caching reduces remote accesses ⇒ reduces server load
• Server disk throughput?– optimized better for large requests than random disk blocks
• Data integrity?– cache-consistency problem due to frequent writes
• Operating system complexity?– simpler for remote service.
83
Four places to cache files
q Server’s disk: slow performanceq Server’s memory
v cache management, how much to cache, replacementstrategy
v still slow due to network delayq Client’s disk
v access speed vs server memory?v large files can be cachedv supports disconnected operation
q Client’s memoryv fastest accessv can be used by diskless workstationsv competes with the VM system for physical memory
space
84
Cache consistency
v Reflecting changes to local cache to master copyv Reflecting changes to master copy to local caches
update/invalidate
Copy 1
Copy 2
Master copy
write
85
Common update algorithms for client caching
q Write-through: all writes are carried out immediatelyv Reliable: little information is lost in the event of a client crashv Slow: cache not useful for writes
q Delayed-write: writes do not immediately propagate to serverv batching writes amortizes overheadv wait for blocks to fillv if data is written and then deleted immediately, data need not
be written at all (20-30 % of new data is deleted with 30 secs)q Write-on-close: delay writing until the file is closed at the
clientv semantically meaningful delayed-write policyv if file is open for short duration, works finev if file is open for long, susceptible to losing data in the event of
client crash
86
Cache coherence
q How to keep locally cached data up to date / consistent?q Client-initiated approach
v check validity on every access: too much overheadv first access to a file (e.g., file open)v every fixed time interval
q Server-initiated approachv server records, for each client, the (parts of) files it
cachesv server responds to updates by propagation or invalidation
q Disallow caching during concurrent-write or read/writesharingv allow multiple clients to cache file for read only accessv flush all client caches when the file is opened for writing
87
NFS – server caching
q Readsv use the local file system cachev prefetching in UNIX using read-ahead
q Writesv write-through (synchronously, no cache)v commit on close (standard behaviour in v4)
88
NFS – client caching (reads)
q Clients are responsible for validating cache entries(stateless server)
q Validation by checking last modification timev time stamps issues by serverv automatic validation on open (with server??)
q A cache entry is considered valid if one of the followingare true:v cache entry is less than t seconds old (3-30 s for files,
30-60 s for directories)v modified time at server is the same as modified time on
client
89
NFS – client caching (writes)
q Delayed writesv modified files are marked dirty and flushed to server on
close (or sync)
q Bio-daemons (block input-output)v read-ahead requests are done asynchronouslyv write requests are submitted when a block is filled
90
File sharing semantics
q Semantics of File sharingv (a) single processor gives sequential consistencyv (b) distributed system may return obsolete value
91
Consistency semantics for file sharingq What value do reads see after writes?q UNIX semantics
v value read is the value stored by last writev writes to an open file are visible immediately to others with the
file openv easy to implement with one server and no cache
q Session semanticsv writes to an open file are not visible immediately to others with
the file opened alreadyv changes become visible on close to sessions started later
q Immutable-Shared-Files semantics - simple to implementv A sharable file cannot be modifiedv File names cannot be reused and its contents may not be
alteredq Transactions
v All changes have all-or-nothing propertyv W1,R1,R2,W2 not allowed where P1 = W1;W2 and P2 = R1;R2
92
NFS – file sharing semantics
q Not UNIX semantics!q Unspecified in NFS standardq Not clear because of timing dependenciesq Consistency issues can arise
v Example: Jack and Jill have a file cached. Jack opens thefile and modifies it, then he closes the file. Jill thenopens the file (before t seconds have elapsed) andmodifies it as well. Then she closes the file. Are bothJack’s and Jill’s modifications present in the file? Whatif Jack closes the file after Jill opens it?
q Locking part of v4 (byte range, leasing)