File Management

Operating Systems I

Introduction

The File Manager controls every file in the system. We’ll learn: How files are organized logically. How they are stored physically. How they are accessed. Who is allowed to access them.

We’ll also study the interaction between the File Manager and the Device Manager.

Introduction - 2

File Manager’s efficiency depends on: How files are organized:

sequentialdirectindexed sequential

How records are structured:fixed length recordsvariable length records

How access to files is controlled

The File Manager

The file management system is responsible for creating, deleting, modifying, and controlling access to files.

The file management system works in conjunction with the Device Manager.

File Manager Responsibilities

Keep track of where files are stored.Determine where/how files are

stored: use space efficiently and provide efficient access to files.

Allocate each file when access to it is cleared, and record use.

File Manager Responsibilities

Deallocate the file when it is to be returned to storage, and communicate its availability to others who may be waiting to use it.

Analogy: File System is like a library ...

File Manager Responsibilities

The File System is like a library with the File Manager playing the role of librarian who performs the same four tasks: Use card catalog to track items and to

help patrons find the book. Predetermined policy used to store all

items in the collection, even over-sized items, where they can be easily found.

File Manager Responsibilities

Library analogy (continued): When the book is requested, it is

retrieved from the shelf and the borrower’s name is recorded in the circulation file.

When the item is returned, the entry in the circulation file is deleted and the item is reshelved.

File Manager Responsibilities

The File Manager keeps track of files in directories, which contain the file name, physical location in secondary storage, and other details about the file.

Predetermined policy determines where each file is stored and how it is accessed.

File Manager Responsibilities

Policy must also determine who has access to what material. Two factors are involved: Flexibility of access to information Protection of information

File Manager allows access to shared files, provides distributed access, and permits browsing of public directories.

File Manager Responsibilities

Meanwhile, the operating system must protect its files against system malfunctions and provide security checks.

The system allocates a file by loading it from secondary storage into memory and updating its records of who is using the file.

File Manager Responsibilities

The File Manager deallocates a file by updating the file tables, and writing the file (if necessary) to secondary storage.

Any processes waiting for the file are notified of its availability.

Definitions

Field - a group of related bytes that can be identified by the user with a name, type, and size. A record is a group of related fields.

File - a group of related records that contains information to be used by specific applications.

Definitions - 2

Database - appears to File Manager as a type of file but is actually more complex. A database is typically a group of related files interconnected at various levels to provide very flexible access to the data. The File Manager must support any specific file structure required.

Definitions - 3

Program files contain instructions; data files contain data. File Manager treats them the same way.

Directories are lists of file names and their attributes. File Manager treats them in a way similar to files.

Every bit of system information is contained in a file!

Interacting With the File Manager

Interaction with the File Manager comes in two ways: User commands issued interactively Commands imbedded in programs

Examples of interactive commands include CREATE, DELETE, RENAME, and COPY.

Examples of imbedded commands include OPEN, CLOSE, READ, WRITE, MODIFY.

Interacting With the File Manager - 2

OPEN and CLOSE pertain to making the file available to the program. OPEN requests allocation to the program.

Specifies a physical and/or logical file name.Often specifies whether file is used for input or

output (or both) by the program.

CLOSE deallocates a file from the program.No more communication with the file unless the

file is OPENed again.

Interacting With the File Manager - 3

The user interface commands are designed to be as simple as possible - they are device independent. No need to know where the file is on the

storage device. In fact, we need not even know what type

of device the file is on.Each logical command translates to a

series of physical commands to a device.

Interacting With the File Manager - 4

For example, when a program issues a command to READ a record from a file on a movable-head disk, the READ command translates to something like this: Move read/write head to cylinder

containing the record. Wait for the required sector to pass under

the disk head (rotational delay).

Interacting With the File Manager - 5

READ command (continued): Activate the appropriate read/write head and

read the sector containing the record. Transfer the sector contents to main memory.

(Program deals with logical records.) Send a flag to indicate the device is free for

another request.

While all this is happening the operating system is checking for error conditions.

Interacting With the File Manager - 6

Without the File Manager these low-level instructions would have to be included in every single program.

In fact, low-level commands for every device used would have to be included: keyboard, monitor, mouse, printer, etc.

The instructions would have to be specific for each manufacturer and model.

Volume Configuration

Each storage unit on a system - removable or not - is considered a volume. Volumes generally contain many files - thus the term “multifile volumes.”

In some cases, very large files may require several volumes - these are called “multivolume files.”

Volume Configuration - 2

Each volume is generally given a name (just as files are). This and other information about the volume is generally written in an easily-accessible location on the volume - at the beginning of magnetic tape or in the first sector of the outermost track of the disk.

Volume Descriptor

Creation Date

Pointer to Directory Area

File System Code

Pointer to File Area

Volume Name

Date volume was created

First sector containing directory

First sector containing normal file

Detects improper volume formats

User-assigned name

Volume Configuration - 3

The master file directory (MFD) is stored immediately after the volume descriptor - it lists the names and descriptions of every file contained on the volume.

Subdirectories are also listed here.The rest of the volume is used for

storage.

Volume Configuration - 4

Early operating systems contained only a single directory per volume. It was organized by name, file location, or chronologically.

This scheme was easy to implement and maintain but it had some major disadvantages:

Volume Configuration - 5

Disadvantages of a single directory per volume: Lengthy search time required,

especially if the directory is organized arbitrarily.

Directory space could fill up before the volume does if many small files are stored on the volume.

Users could not create subdirectories to group related files.

Volume Configuration - 6

Disadvantages of a single directory per volume (continued): Multiple users could not protect their

files from being seen - the entire directory was listed on request.

Each file or program on the volume required a unique name no matter how many people used the system. For example, only one file named PROG1.EXE could be on the volume.

About Subdirectories

Most file systems support “user directories” or subdirectories. A subdirectory is treated as a file but its MFD entry is flagged so that the File Manager knows this “file” is really a directory. Early implementations of subdirectories created one per user.

Most modern systems allow users to create their own subdirectories.

Directory Tree Structure

Modern systems use an inverted tree structure for directories, with the root being the MFD of the volume.

See Figure 8.2, page 186, or …Simply open Windows Explorer on a

Windows-based computer for a graphical representation of the directory structure.

Subdirectories (Continued)

Regardless of the complexity (or simplicity) of the directory tree on a volume, each file entry in every directory contains a file descriptor which contains information about the file including: File name File type (for example, directory, normal file) File size - kept here for convenience

Subdirectories (Continued)

File descriptor information (continued): File location - first physical block where the

file is stored Date and time of creation Owner Security information - who can access the

file and what type of access (e.g., read only) is allowed

Record size - fixed or maximum record size

File Naming Conventions

The complete file name of a file may be much longer than it appears to be.

Every file has a relative file name and an extension.

We will use the term absolute file name for the complete file name. The short name will be called “relative name.”

Naming Conventions - 2

The relative (short) name is chosen by the user when the file is created. It’s typically 1-12 characters in length (note: some systems allow file names up to about 255 characters) and contains letter, numbers, and special characters. (Each operating system has its own set of rules.)

The name is followed by a period and an extension.

Naming Conventions - 3

The extension identifies the type of file or its contents. (For example, this “slide show” is created using Microsoft’s PowerPoint software - identified by the extension “ppt” after the relative (short) name - class7. The absolute file name is class7.ppt in this case.

Many extensions are “built-in” to the OS.

Naming Conventions - 4

Extensions like .BAT, .TXT, .BAS, etc., are recognized by the operating system as associated with a specific program. Others are included with the installation of certain software packages (like Microsoft Office or Microsoft PowerPoint in the case of PowerPoint presentations).

Of course, users can establish their own.

Naming Conventions - 5

The additional information required for a complete file name depends on the operating system. For example: For a DOS or Windows user on a PC: C:\

PARTS\INVENTRY.FOR In a networked VAX environment with VMS:

VAX2::USR3:[IMFST.FLYNN]INVENTRY.FOR;7

In a UNIX environment:/usr/imfst/flynn/inventry.for

Naming Conventions - 6

The more flexible the File Manager, the more information needed for the complete file name. Some shortcuts can simplify this task, though: Typical systems will look first in the current

directory. Many systems specify a search PATH for

files without complete names. “Browse” functionality in Windows

systems.

File Organization

Record Format - Records in a file can be fixed-length or variable-length, blocked or unblocked.

Fixed-length records are most common, easiest to access, and ideal for use in data files. If the record length is too short, data is truncated. If it’s too long, space is wasted.

File Organization - 2

Variable-length records do not truncate data or waste storage space. They are easy to access if read one after the other (sequentially) but direct access is difficult.

For that reason, variable-length records are used most often in sequential files like text files or programs, or when records are indexed.

File Organization - 3

Physical file organization has to do with how the records are arranged and the physical medium used to store it.

On magnetic disk, records in a file can be arranged in one of three ways: sequential direct access indexed sequential

File Organization - 4

In considering the organization of the file the programmer or analyst looks at: Volatility of the data - the frequency with

which additions and deletions are made. Activity - percentage of records processed

during a given run. Size of the file. Response time - how long the user is

willing to wait for a request to complete.

File Organization - 5

Sequential record organization - records are stored one after the other and retrieved in the same way. To speed things up, records may be sorted by a key field; searches look for the record containing a matching key value. Sorted order must be maintained. Rewrite of file necessary to insert new

records in the middle of the file.

File Organization - 6

Direct record organization (or direct access) allow access to records in any order. Records are referenced by relative

address (or logical address) which is computed when the record is stored and again when it is retrieved.

Key field is used in a hashing algorithm to produce a logical address for the record.

File Organization - 7

Direct record organization (continued): Logical address is transformed into a

physical address. Same procedure is used for record

retrieval.Hashing algorithms will not necessarily

produce a unique address for each record! (That adds complexity.)

File Organization - 8

Indexed sequential record organization combines the best of sequential and direct access. No hashing algorithm. Uses an index file through which

records are retrieved. Index file entries contain a key and the

physical block which contains this and lower keys.

File Organization - 9

Indexed sequential organization (continued): Record access: system begins by

searching the index file. Once the key is found, the physical block of the data record is accessed.

Overflow areas are spaced throughout the file.

Little or no programmer effort involved in maintaining or organizing the file.

File Organization - 10

Indexed sequential organization (continued): Once the overflow area becomes too

large, the file needs to be reorganized because access is much slower.

Typical method: copy the old file to a new one, delete the old one and rename the new file (to the old name).

Physical Storage

File Manager must work with files as logical units or records as well as files as a whole.

Records within a file must all have the same format but may have varying lengths.

Figure 8.3 illustrates this ...

Fixed-length Records

R1 R2 R3 R4 R7R6R5

Unblocked, fixed-length records

R1 R2 R3 R4 R5 R6 R7

Blocked, fixed-length records

Block

Variable-length Records

R1R2

Length

Unblocked, Variable-length records (length precedes record)

Block 1Length

R1Length

R1R2

LengthR2

Blocked, Variable-length records

Block

R1Length

R2

File Storage

“File storage” actually refers to record storage - how the records are stored physically.

Contiguous storage - refers to all file records being stored one after the other in one continuous block of storage. Simple to implement and manage. Directory very streamlined.

Contiguous Storage

Contiguous storage disadvantages: A file cannot be expanded unless empty

space is available following it. Files must be moved to a new location if

no room for expansion - leads to fragmentation of disk.

Noncontiguous Storage

Noncontiguous storage allows allocation of files anywhere there is space available on the disk. Files stored contiguously if possible. Additional records stored on other parts

of the disk (these are sometimes called file extents).

Extents are linked together by pointers at the storage level or directory level.

Linking - Storage Level

Linkage - Directory Level

Indexed Storage

Indexed Storage allows direct access by bringing all the file extent pointers into an index block.

Each file has its own index block.Index entries are in the order of file

extents: first index entry = first file extent, second index entry is for extent 2, etc.

Indexed Storage

Indexed Storage

Index pointers are set to null when a file is created. As extents are created, the pointer in the index block is set to the sector containing the extent.

This scheme supports both sequential and direct access.

Because each file requires an index block, it doesn’t improve storage utilization.

Data Compression

Used to save space in files.Three methods for compressing data

in databases: Records with repeated characters -

repeated characters replaced with a code indicating how many characters were (such as blanks) were truncated. For example, “Adamsbbbbbbbbbb” becomes “Adamsb10”

Data Compression - 2

Numbers with many zeroes can be shortened using a code to indicate how many zeroes were at the end of the original number. For example, the number 300000000 becomes 3#8.

Repeated terms can also be compressed. Symbols are used to replace commonly-used words in a database.

Data Compression - 3

Front-end compression is used in database management systems for index compression.

For example, an alphabetical list of student names could be compressed as shown in Table 8.1 (see next slide …)

Front-End Compression

Original list Compressed ListSmith, Betty Smith, BettySmith, Gino 7GinoSmith, Donald 7DonaldSmithberger, John 5berger, JohnSmithbren, Ali 6ren, AliSmithco, Rachel 5co, RachelSmither, Kevin 5er, KevinSmithers, Renny 7s, RennySnyder, Katherine 1nyder, Katherine

Access Methods

Dictated by a file’s organization. Indexed Sequential organization offers

the most flexibility. Sequential organization offers the least.

A sequential file can only support sequential access to its records. Current byte address (CBA) must be

updated with each access.

Sequential Access

For sequential access of fixed-length records, the CBA is updated by incrementing it by the record length (RL): CBA = CBA + RL.

For access of variable-length records, the CBA is updated by adding the record length plus the number of bytes used to hold the length: CBA = CBA + RL + N.

Direct Access

For direct access of fixed-length records, the CBA can be computed directly from the record length and the desired record number (RN): CBA = (RN - 1) * RL.

For direct access of variable-length records, computing the CBA is virtually impossible. (An alternative is to simply do a sequential search.)

Indexed Sequential Files

These files offer either sequential or direct access. The CBA is computed using either method already seen, with one exception: direct access is through the index.

Index is normally much smaller than the file itself and could be kept in main memory for quicker searches.

Levels in a File Management System

Efficient file management cannot be separated from efficient device management.

A wide range of functions have to be organized for an I/O system to perform well.

Figure 8.9 (see next slide) shows one such hierarchy.

I/O System Hierarchy

Basic File System

Access Control Module

Logical File System

Physical File System

Device Interface Module

Device

Levels in a File Management System - 2

The access control verification module verifies the user’s permissions to perform the requested operation on the file. If not allowed, an “access denied”

message is sent to the user. Otherwise, control passes on to the

logical file system.

Levels in a File Management System - 3

The logical file system uses information passed down by the basic file system to compute the CBA from the record number: CBA = (RN - 1) * RL.

This result, along with the address of the first physical record (and block size, if records are blocked) are passed to the physical file system.

Levels in a File Management System - 4

The physical file system computes the location of the desired record.

This information is passed on to the device interface module, which transforms the record location to a physical cylinder / surface / record combination.

Levels in a File Management System - 5

Information is copied from the storage device into a buffer in memory.

The “all clear” signal is sent to free the device for the next request.

A WRITE works the same way as a READ except that the allocation module is called by the device interface module to handle allocation of free space to a file.

Access Control Verification Module

Early operating systems did not support file sharing: there might be ten duplicate copies of a compiler, for example.

The advantages of file sharing are readily apparent: space savings, synchronization of updates, and sharing of data in main memory.

However, progress brings problems ...

Access Control Verification Module - 2

Disadvantages of file sharing: Integrity of each file must be safeguarded

Requires control over who can do what with the file

Users can perform these actions: READ, WRITE, DELETE, EXECUTE, or some combination of those four.

Each file management system has its own method of controlling file access ...

Access Control Verification Module - 3

Controlling file access: four methods are common: Access control matrix Access control lists (ACLs) Capability lists Lockword control

Access Control Matrix

The access control matrix is easy to implement: Works well only for systems with a few

files and a few users. Each column identifies a user. Each row identifies a file. Intersection identifies user’s access

rights to the file.

Access Control Matrix

Access Control Lists

The access control list is a modification of the access control matrix. Each file is entered in the list.

Each entry contains the names of users allowed to access the file and their access permissions.

Only users with permission to use the file are listed. Others are grouped under a global heading such as WORLD.

Some systems group users into groups such as SYSTEM, OWNER, GROUP, and WORLD.

Access Control Lists - 2

SYSTEM - the operating system and administrators.

OWNER - the account used to create the file.

GROUP - Owner may set up a group of people who have access to the file.

WORLD - Everyone who does not fall into one of the other groups.

Access Control Lists - 3

File Access

File 1 USER1 (RWED), USER2 (R-E-), USER4 (RWE-),USER5 (--E-), WORLD (----)

File 2 USER2 (R-E-), USER3 (R-E-), USER4 (--E-), WORLD (----)

File 3 USER2 (RWED), USER4 (--E-), WORLD (----)

File 4 USER1 (R-E-), USER5 (RWED), WORLD (----)

File 5 USER5 (RWED), WORLD (----)

Capability Lists

A capability list shows the access control information from the point of view of the user.

For each user, the list shows the access permissions that user has.

The capability list is gaining in popularity (access control lists are most common) since it can also control access to devices.

A Capability List

User Access

USER1 File1 (RWED), File4 (R-E-)

USER2 File1 (R-E-), File2 (R-E-), File3 (RWED)

USER3 File2 (R-E-)

USER1 File1 (RWED), File4 (R-E-)

USER4 File1 (RWE-), File2 (--E-), File3 (--E-)

USER5 File1 (--E-), File4 (RWED), File5 (RWED)

Lockwords

Lockwords protect files much like a password protects access to a system. (In some system, the term password is used in the same context as lockword.)

The lockword is stored in the directory but not shown with the directory listing. It has the advantage of requiring little storage.

Lockwords - 2

Anyone who wants access to the file must supply the correct lockword to gain access.

Disadvantages of lockwords include: May be guessed by hackers. Lockwords do not control type of access

to the file. Anyone who knows the lockword has full access to the file - even if that was not the intent of the file owner.

Conclusion

The File Manager controls every file in the system and processes the user’s commands to interact with the files. It also manages the access control procedures to maintain the integrity and security of files under its control.

The File Manager must be able to accommodate many file organizations.

Conclusion - 2

The File Manager must also accommodate a variety of physical storage allocation schemes, record types, and access methods. This requires increasingly complex file management software.

We covered in this chapter: Sequential, direct, and indexed sequential

file organization;

Conclusion - 3

We covered in this chapter (continued): Contiguous, non-contiguous, and indexed

file storage allocation; Fixed- versus variable-length records; Four methods of access control.

We need to realize the strengths and weaknesses of each segment of the File Manager in order to get the most from it.