CIS 620 Advanced Operating
Systems
Lecture 8 – Naming and File SystemsProf. Timothy Arndt
BU 331
Names, Identifiers, And Addresses
• Naming was originally (and still is primarily, to some extent, today) applied to file systems Nowadays other entities are organized into
namespaces• Processes, IPCs, etc.
• Properties of a true identifier:• An identifier refers to at most one entity.• Each entity is referred to by at most one identifier.• An identifier always refers to the same entity
Flat Naming
• Non-hierarchical Can locate an entity using flat naming by
broadcasting in a LAN (e.g. ARP) In a point-to-point network (WAN) can use
multicasting What about flat naming for mobile entities?
• Can use the forwarding pointers approach
Forwarding Pointers
• The principle of forwarding pointers using (client stub, server stub) pairs.
Forwarding Pointers
• Redirecting a forwarding pointer by storing a shortcut in a client stub.
Forwarding Pointers
• Redirecting a forwarding pointer by storing a shortcut in a client stub.
Home-Based Approaches
• The principle of Mobile IP. A home location keeps track of the current location an entity
Distributed Hash Tables
• Mapping a name to a location in a DHT can take a long time if we take the naïve approach (just try each host around the logical ring until we find the name) Instead, we can use a finger table which
records locations of entities at logarithmically increasing distances
Distributed Hash TablesGeneral Mechanism
• Resolving key 26 from node 1 and key 12 from node 28 in a Chord system.
Name Spaces
An alternative to flat naming is structured naming leading to a name space
A general naming graph with a single root node.
Linking and Mounting
• The concept of a symbolic link explained in a naming graph.
Linking and Mounting
• Information required to mount a foreign name space in a distributed system• The name of an access protocol.• The name of the server.• The name of the mounting point in the foreign
name space.
Linking and Mounting
• Mounting remote name spaces through a specific access protocol.
Name Space Distribution
• An example partitioning of the DNS name space, including Internet-accessible files, into three layers.
Name Space Distribution
• A comparison between name servers for implementing nodes from a large-scale name space partitioned into a global layer, an administrational layer, and a managerial layer.
Name Resolution
• Discovering the location of a resource, given its name, is called name resolution Name resolution in hierarchical systems like
DNS uses iteration, recursion (and caching)
Implementation of Name Resolution
• The principle of iterative name resolution.
Implementation of Name Resolution
• The principle of recursive name resolution.
Implementation of Name Resolution
• Recursive name resolution of <nl, vu, cs, ftp>. Name servers cache intermediate results for subsequent lookups.
Example: The Domain Name System
• The comparison between recursive and iterative name resolution with respect to communication costs.
The DNS Name Space
• The most important types of resource records forming the contents of nodes in the DNS name space.
DNS Implementation
• An excerpt from the DNS database for the zone cs.vu.nl.
DNS Implementation
• An excerpt from the DNS database for the zone cs.vu.nl.
DNS Implementation
• Part of the description for the vu.nl domain which contains the cs.vu.nl domain.
The File System in UNIX
The UNIX file system is organized in a tree structure. The files on this tree are one of the following types:• An ordinary file• A directory file• A special file (representing an I/O device)• A link that points to another file• A socket that is used for interprocess
communication
Directories
A directory is a file whose content consists of directory entries for the file in the directory.• A directory entry contains the name of the file and a
pointer to the file. The pointer is an index called the i-number to a table known as the i-list.
• Each entry in the i-list is an i-node containing status and address information about a file or is free.
• The entire file system may contain several self-contained filesystems, each with its own i-list.
Directories
grail:/users/faculty/arndt> df/home0 (arthur.cba:/home0 ): 14736728 blocks -1
i-nodes/home1 (arthur.cba:/home1 ): 7846106 blocks -1
i-nodes/userspace (shamu.cba:/export/home0/userspace): 1327190 blocks -1 i-nodes/usr (/dev/vg00/lvol3 ): 65700 blocks 13437
i-nodes/tmp (/dev/vg00/lvol4 ): 219084 blocks 12016
i-nodes/users (/dev/vg00/lvol5 ): 245610 blocks 65662
i-nodes/mnt1 (/dev/vg00/lvol6 ): 129708 blocks 17093
i-nodes/client_server (/dev/dsk/c4d0s2 ): 512710 blocks 357634
i-nodes/ (/dev/vg00/lvol1 ): 23880 blocks 6769
i-nodes
grail:/users/faculty/arndt>
Special Files
UNIX represents I/O devices such as terminals, printers, tape, and disk drives as special files in the file system.• In this way, an application program can treat file and
device I/O in the same way.• Each special file is stored in the directory /dev.• A special file can be either a character special file or
a block special file. A character special file represents a character-oriented I/O
device. A block special represents a high-speed I/O device that
transfers data in blocks rather than bytes.
Links
A directory entry may be a pointer to another file. This is called a link. There are two kinds of links:• hard links• symbolic links
A hard link is an entry in a directory with a name and some other file’s i-number. The hard link is not distinguishable from the original file.• A file may have several links to it. The link must be
to an ordinary file in the same filesystem.
Links
A link is created to another file by using the ln command:ln fileln file linknameln file1 . . . Dir
By default, a hard link is created by ln. • It is not necessary to be the owner of a file to link to
the file. The command rm removes the directory entry of the file. The file is physically deleted when the last link to it is removed.
• The number of hard links to a file is kept as part of the status.
Symbolic Links
A symbolic link is a directory entry that contains the pathname of another file. The symbolic link can be used as an argument in a command.
The file pointed to can be removed and the link will remain.
A symbolic link can span filesystems and may point to a directory.
A symbolic link can be created using ln -s
File Access Control
• The file mode for a file as displayed by ls -l is, for example:-rw-rw-rw- The first position shows the file type: - for
normal, d for directory, l for symbolic link, c for character special file, b for block special file.
Positions 2-4 show read, write, and execute permission for the owner of the file. A - means no permission.
File Access Control
Positions 5-7 show the permissions for users in the file group.
Positions 8-10 show the permissions for other users.• To access a directory, you must have execute
permission for that directory. To list its contents, you must have read permission. Write permission is necessary to create or delete a file in a directory
• New files are created with a default protection. To change this default, use the command umask. This command uses three octal numbers.
File Status
A file status is kept in the i-node for each file in the UNIX files system. The file status include:
mode number of links owner, group size last access, last content change, last status change i-number device block size block count
File Mode
The file mode consists of 16 bits. We have already seen the low order 9 bits: the permissions. The four high bits specify the file type while the next three bits define the way in which an executable file is run.• Set userid: owner of a file changed to user during
execution.• Set groupid: sets group id during execution.• The stickybit keeps a shared binary in memory.
The mode of a file may be changed using the chmod command
File Mode
grail:/usr/bin> ls -l e* | more-r-xr-xr-t 6 bin bin 245760 Sep 7 1993 e-r-xr-xr-t 6 bin bin 245760 Sep 7 1993 edit-rwxr-sr-x 1 root bin 479232 Jun 17 1996 elm-r-xr-sr-x 1 bin mail 270336 Sep 7 1993 elm.orig-r-xr-xr-x 1 bin bin 32768 Jun 17 1996 elmalias-r-sr-xr-x 1 lp bin 20480 Oct 22 1993 enable-r-xr-xr-t 6 bin bin 245760 Sep 7 1993 ex-r-xr-xr-x 1 bin bin 16384 Sep 7 1993 expandgrail:/usr/bin>
Note the ‘s’ and ‘t’ above in place of ‘x’.
Groups
• A user can belong to one or more groups specified in the file /etc/group.
• Use the command ‘groups’ to see which groups you belong to.
• Each file is associated with a group owner. To change the group owner of a file use ‘chgrp’ This allows groups to work on the same files
(but it doesn’t coordinate their changes!).
File System Implementation
A UNIX file is an array of bytes stored in a number of data blocks in a specific filesystem.
Originally, blocks were 512 bytes. Some implementations now use 4K bytes per block. Since the final block may waste space if it is not full, it may be broken up into fragments of (e.g.) 1K bytes.
As said previously, information on files is stored in an i-node, where the i-nodes for a filesystem are stored in an i-list.
File System Implementation
The i-node also contains pointers to the data blocks of the file. An i-node contains 10 pointers to data blocks, a pointer to a data block containing pointers to data blocks, and a pointer to a data block containing pointers to data blocks containing pointers to data blocks (!). This allows for extremely large files.
Why use this arrangement rather than just a chain of data blocks?
Removable Filesystems
• One of the UNIX filesystems is the root filesystems. All of the other filesystems are removable. Removable filesystems are attached to the root
filesystem at some leaf node using the mount command: mount devfile directory [-r]
The command umount is used to unmount a removable filesystem.
Mount as read only
Super Block and Cylinder Groups
The super block of a filesystem records information about the filesystem:• Size of the filesystem• Block/fragment size• Length of the i-node list• A list of free blocks and length of the list• A list of free i-nodes and length of the list
The super block is stored at the beginning of a filesystem.
Super Block and Cylinder Groups
Contemporary UNIX system use an improved organization based on the idea of a cylinder group (a set of physically close cylinders on the disk).
I-nodes are distributed within each cylinder group rather than at the beginning of the filesystem. Therefore, i-nodes are close to their data blocks, and access time is improved. Super blocks are also duplicated in order to increase robustness.
Filesystem Table
Each filesystem is represented by its own block-type special file. The name of this file along with other information is kept in /etc/checklist or /etc/fstab:
/dev/vg00/lvol1 / hfs defaults 0 1 31484 564/dev/vg00/lvol2 swap ignore sw 0 0 16392 592/dev/vg00/lvol3 /usr hfs defaults 0 2 0/dev/vg00/lvol4 /tmp hfs defaults 0 2 13 16408/dev/vg00/lvol5 /users hfs quota,defaults 0 2 0/dev/vg00/lvol6 /mnt1 hfs rw,suid 0 3 0/dev/dsk/c4d0s2 /client_server hfs quota,defaults 0 2 0arthur.cba:/home0 /home0 nfs rw 0 2 0arthur.cba:/home1 /home1 nfs rw 0 2 0shamu.cba:/export/home0/userspace /userspace nfs rw 0 2 0
Filesystem Table
The currently mounted tables are kept in /etc/mnttab:
/dev/vg00/lvol1 / hfs defaults 0 1 917804700 1/dev/dsk/c4d0s2 /client_server hfs quota 0 2 917804709 1/dev/vg00/lvol6 /mnt1 hfs defaults 0 3 917804709 1/dev/vg00/lvol5 /users hfs quota 0 2 917804709 1/dev/vg00/lvol4 /tmp hfs defaults 0 2 917804709 1/dev/vg00/lvol3 /usr hfs defaults 0 2 917804709 1shamu.cba:/export/home0/userspace /userspace nfs rw 0 2
917804764 0arthur.cba:/home1 /home1 nfs rw 0 2 917804764 0arthur.cba:/home0 /home0 nfs rw 0 2 917804764 0
Network File System
Many UNIX systems allow file operations on remote filesystems stored on other computers on a LAN. This is done using NFS.
A machine on a LAN can function as file server (e.g. by having all users home directories on that machine).• NFS can work on different hardware and OSs (e.g.
different varieties of UNIX). A filesystem is made available remotely by
exporting it. It can then be mounted on a remote machine.
Linux File Types
• Besides regular files (-), directory files (d), block special files (b) and character special files (s) Linux also has the following file types Unix domain socket (s) Soft link (l) Named pipe (p)
File System Consistency
In order to improve filesystem performance, a pool of data blocks are kept in memory so that further references to them don’t require a disk access.
Changes made to these blocks are not immediately reflected on disk.
Periodically, they are written to disk. This arrangement is known as lazy write.
• DOS used careful write in which changes are written immediately to disk.
Powering down the computer without going through a shutdown sequence can cause unwritten data to be lost.
File System Consistency
The program fsck can be run when the system is rebooted to check for and repair file system inconsistency (compare Windows chkdisk program).
This loss of data is unacceptable for mission-critical database management systems.• Therefore these systems completely bypass the
UNIX filesystem, managing a raw device by themselves in order to ensure a consistent state of the database in case of system failure.
Problems
• The user types in the command “cat /user/home/me/sub/file1” - what are the steps involved in retrieving the data in the file?
• A removable hard drive formatted for Unix (i.e. Firewire or USB) is attached to a Unix system. What steps are taken in order to make the data on the drive available to application programs?
procfs
• procfs (the proc filesystem) is a special filesystem in Linux and other Unix-like OSs that presents information about processes and other kernel information in a a hierarchical file-like structure Makes it easier to access info about (e.g.)
processes Mounted at /proc
Windows Filesystems
Windows NT supported multiple file system types:• FAT (for compatibility with non-NT systems)• HPFS (the native OS/2 file system)• NTFS (the native NT file system type)
NTFS incorporates the normal features of other modern file systems.• Journaling• Enhanced file security• Lazy-write scheme• Mirrored master file table and boot sector
Journaling
Like AIX, NTFS uses techniques borrowed from real-time transaction processing to ensure file system structure integrity at all times.
Before any change is made to the file system, it is first recorded in a designated log file.
During a system boot, the file system structure is automatically made current with its log.
This doesn’t prevent data loss since only changes to the structure of the file system are logged, not changes to a file’s contents.
Lazy-write scheme
As in Linux, disk operations are performed to a cache in memory rather than directly to disk.
• This optimizes system I/O performance, but may result in file changes being lost in case of a power failure.
Windows also uses the lazy-write scheme on FAT file systems, unlike DOS.
The journaling technique allow for a quicker recovery than other lazy-write systems - chkdsk is not required to scan the disk in case of crash.
Master File Table
The equivalent of the i-list in NTFS is the Master File Table (MFT)• The first 16 records in the MFT contains metadata
files used to implement the file system: Master File Table: a list of all contents of the NTFS
volume MFT2: a mirror of the first three records of the MFT, used
to guarantee access to the MFT in case of a single-sector error
Log File: a list of transaction steps used for recoverability. Volume: the volume name, NTFS version, and other info. Attribute Def. Table: A table of attribute names, numbers,
and descriptions
File System Consistency
Root Filename Index: Root folder Cluster Bitmap: A representation of the clusters showing
which clusters (i.e. blocks) are in use. Partition Boot Sector: the bootstrap for the volume, if
bootable. Bad Cluster File: List of bad clusters. Quota table: Disk quota usage for each user on a volume.
Currently unused. Upcase Table: Used for converting lowercase characters to
the matching Unicode uppercase characters. The next 5 MFT records are reserved for future use.
MFT Records
A file in an NTFS file system is a collection of attributes stored in an MFT record (and possibly elsewhere on the disk). Attributes include the following:
Basic file information, such as the file’s DOS attributes, modification and access time, and number of hard links.
Filename: NTFS filenames may be up to 255 characters in length and may contain any letters, number or Unicode characters. A DOS-style filename may also be given.
Security attributes: the Windows access permissions. Data attribute: the data in the file. Additional indexing data for directories.
Advanced Filesystem Features
NTFS has optional transparent file compression.
On Windows, all local file systems are automatically mounted when the the system boots.
File system volumes (as defined with a unique drive letter) can occupy more than one disk partition.
Striped file systems may be created for higher I/O performance.
Advanced Filesystem Features
RAID filesystems are also available. RAID (Redundant Array of Inexpensive
Disks) is a set of six specific disk organizations that can be used to improve disk performance, or reliability, or both.• Most RAIDs consist of a RAID SCSI controller plus
a box of SCSI disks that appear to the OS as a single large disk. No software changes are required to use the RAID.
RAID
The data in a RAID are distributed over the drives, to allow parallel operation. Several schemes were devised by Patterson and are known as RAID level 0 through RAID level 5.
RAID level 0 stripes data over multiple drives in round robin fashion. In this way, a block of data can be read from multiple drives, working in parallel. This works best when data is requested in large chunks. The controller must coordinate the request and issue the individual requests to the disks.
RAID
RAID level 0 is less robust than using a single disk, since the failure of any disk leads to failure for the RAID.
RAID level 1 is a true RAID. It duplicates all the disks so that there are four primary and four backup disks. On a write, a strip is written twice. On a read, either copy can be read. Fault tolerance is excellent.
RAID
RAID levels 4 and 5 work with strips of data, with parity. RAID level 4 writes a parity strip (computed from the strips on each of the other disks) on an extra drive. If a drive crashes, the lost bytes can be recomputed from the parity drive.
RAID level 5 distributes the parity bits uniformly over all the disks so that the parity drive does not become a bottleneck. This makes recovery more complex, however. Level 5 RAID is supported on Windows Server only.
RAID
Other RAID levels have been defined as well RAID 6
• Identical to RAID 5, except it has a 2nd (distributed) parity block.
RAID 10• Contains a top-level RAID-0 array (or stripe set)
composed of two or more RAID-1 arrays (or mirrors).
Removable Disks
• FAT and NTFS file systems may be created on removable disks (USB external drives, USB flash drives, etc.).
• Such disks are removable in the normal manner when the FAT file system is used.
• For disks formatted with the NTFS system, must use ‘Safely Remove Hardware’ since write caching is used.
Network Filesystem Access
Files and directories can be shared across a network using three mechanisms:• Windows shares: the native facility for sharing disk
resources within a Microsoft network.• The Distributed File System: similar to NFS,
allows disk resources of different Windows computers to be combined into a single, network-accessible directory tree.
• NFS and SAMBA: third party resources allow the sharing of file systems resources between NT and UNIX systems.
File Types
There are several different NTFS file types:• Regular files• Directories• Hard links• Shortcuts (similar to symbolic links, but used in the
GUI environment) Note that there are fewer file types than in Unix
since I/O devices, sockets are implemented differently.
File Ownership and Permissions
• FAT file systems offer virtually nothing in the way of file and directory access controls. NTFS on the other hand, offers sophisticated controls. Every file in an NTFS file system has an owner
associated with it. There is no group owner. Access control lists (ACLs) are used to grant
and limit access to files and directories. ACLs are composed of name-permission pairs
known as access control entries (ACEs).
File Ownership and Permissions
Windows uses the following basic permissions: Read (R), Write (W), Execute (X), Delete (D), Change Permissions (P), Take Ownership (O). There are also named commonly used subsets of these permissions.
It is easier to assign permissions to groups, rather than individual users.• ACLs can be viewed using the security tab of the
file’s Properties dialog box.
Attribute-Based Naming
• In some applications it is useful to be able to lookup entities in terms of (attribute, value) pairs This is called attribute-based naming Implemented by directory services
• Full database approach is too costly• A popular hierarchical implementation is LDAP
Based on the OSI X.500 directory service
Hierarchical Implementations: LDAP
A simple example of an LDAP directory entry using LDAP naming conventions.
Hierarchical Implementations: LDAP
• Part of a directory information tree.
Hierarchical Implementations: LDAP
• Two directory entries having Host_Name as RDN.
Decentralized Approach
• We can also implement directory services in a P2P network, resulting in a decentralized implementation Each entity is described by a hierarchically
organized set of attributes known as an AVTree Hash each path in each tree and assign to DHT
node Query represented by AVTree
• Compute the hashes – send to nodes
Mapping to Distributed Hash Tables
• A general description of a resource. Its representation as an AVTree.
Mapping to Distributed Hash Tables
• The resource description of a query. Its representation as an AVTree.
Semantic Overlay Networks
• Maintaining a semantic overlay through gossiping.