9/9/19
1
T4-Input/Output
1.2
License
This document is under a licenseAttribution – Non-commercial - Share Alike
under the same Creative Commons 3.0 license.To see a summary of the license terms, visit:
http://creativecommons.org/licenses/by-nc-sa/3.0/deed.en
9/9/19
2
1.3
Contents
■ Basic concepts of I/O■ Devices: virtual, logical, physical ■ Linux I/O management ■ Kernel data structures ■ Basic system calls ■ Examples ■ File system ■ Relationship between system calls and data structures
1.4
BASICS CONCEPTS OF I/O
9/9/19
3
1.5
What’s I/O?
■ Definition: information transfer between a process and the outside.● Data Input: from the outside to the process● Data Output: from the process to the outside
(always from the process point of view)
■ In fact, basically, processes perform computation and/or I/O■ Sometimes, even, I/O is the main task of the process:
for instance, web browsing, shell, word processor
■ I/O management: Device (peripherals) management to offer an usable, shared, robust and efficient access to resources
1.6
I/O Devices
9/9/19
4
1.7
HW view : Accessing physical devices
Control Register
State Register
Data Register
Controller PeripheralPorts
Bus
Mem
ory
I/O Bus
CPU
mov ds:[XX], axin ax, 10hout 12h, ax
int
1.8
User view (till now) (PRO1)
Input: cin. Read data, process them in order to adapt them to the data typeOutput: cout. Process data and write them
9/9/19
5
1.9
In this course we’ll see what’s in the middle
write(stdout,p,num_bytes)
read(stdin,p,num_bytes)
• No so easy as in C or C++• Data conversion are up to the user (programmer)
(that’s the duty of C and C++ libraries)
1.10
DEVICES: VIRTUAL, LOGICAL, PHYSICAL
9/9/19
6
1.11
■ To interact with users: display, keyboard, mouse. To store data: hard disk, Bluray, pendrive. To transfer data: modem, network, WI-FI or even more specialized (plane controllers, sensors, robots) … many possible characterizations
■ Classification criteria:● Device type: logical, physical, network● Access speed: keyboard vs hard disk● Access flow: mouse (byte) vs DVD (block)● Access exclusivity : disk (shared) vs printer (dedicated)● And so on …
■ Trade-off: standardization vs new device types
Device types
CONCEPT: Device independence
1.12
■ The goal is that processes (code mainly) be independent of the device that is being accessed
■ Uniform I/O operations ● Access to any device with the same system calls● Increase portability and simplicity of user’s processes
■ Use of virtual devices● Process does not specifies the physical device, but it uses an identifier
and a later translation. ■ Device redirection: use different devices with no code changes
● The Operating System allows a process to change the allocation of its virtual devices
Independence: principles of design
% programa < disp1 > disp2
9/9/19
7
1.13
■ Hence, usually, design in three levels: virtual, logical and physical■ First level gives us independence, the process works on virtual devices
and does not need to know what’s behind. ■ The second level gives us device sharing. Different concurrent accesses to
the same device.■ Therefore, it’s possible to write programs performing I/O on (virtual) devices
without specify which (logical) ones ■ In execution time, process dynamically determines on which devices it is
working. It can be a program argument or “inherited” from its parent.■ Third level separates operations (software) from implementation. This code
is quite low-level, most of times in assembler.
Independence: design principles
1.14
■ Virtual level: isolates user from the complexity of managing physical devices.● It sets correspondence between symbolic name (filename) and user
application, using a virtual device. 4 A symbolic name is the representation inside of the Operating
System– /dev/dispX or .../dispX
4 A virtual device represents a device in use by a process– Virtual Device = channel = file descriptor. It is a number– Processes have 3 standard file descriptors
» Standard Input file descriptor 0 (stdin)» Standard Output file descriptor 1 (stdout)» Standard Error descriptor 2 (stderr)
● I/O System calls use the identifier of the virtual device
Virtual Device
9/9/19
8
1.15
■ Logical level:● It sets correspondence between virtual device and physical(?) device● It manages devices with or without physical representation
4 For instance, a virtual disk (on memory) or a null device● It deals with independent size data blocks● It brings a uniform interface to physical level● It offers shared access (concurrent) to physical devices that represents
(if so)● In this level permissions, such as access rights, are checked.● Linux identifies a logical device with a file name
Logical Device
1.16
■ Physical Level: implements logical level operations in low-level.● Translates parameters from the logical level to specific parameters
4 For instance, on a hard disk, translates file offset to cylinder, platter, sector and track
● Device initialization. Check if it is free, otherwise it enqueues a request● It performs the request programming operation
4 It could mean state checking, hard disk engine init, ...● Waits (or not) the operation ending● If successful, return the results or report any possible error● In Linux, a physical device is identified by three parameters:
4 Type: Block/Character4 And two numbers: major/minor
– Major: tells the kernel which family device to use (DD)– Minor: tells the kernel which one inside the family
Physical Device
9/9/19
9
1.17
■ In order to offer independence, a set of operations is defined for all devices● It is a superset of operations that could be offered to access to a
physical device. ● Not all devices can offer all operations● In translation time (from virtual to logical) the available operations is set
Device Drivers
1.18
■ OS programmers can’t generate code for all devices, models, etc.■ Manufacturers should provide the low-level routines set that implements
device functionality4 The code + data to access device is known as Device Driver4 It follows the interface specification of accessing to I/O operations
defined by the OS■ To add a new device
1. Option 1: with kernel recompilation 2. Option 2: without kernel recompilation
4 OS must offer a mechanism to add kernel code/data dynamically– Dynamic Kernel Module Support, Plug & Play
Device Drivers
9/9/19
10
1.19
■ Common operations (interface) are identified and specific differences are encapsulated inside OS modules: Device Driver.● Isolates the kernel from device management complexity● Protects the kernel from external code
Device Driver
I/O SubsystemProcess
management
Memorymanagement
IPCdevicedriver
kernelnew
devicedriver
Interface to the kernel
insertion into the kernel
Hardware
1.20
■ Device Driver (generic) Routines set that manages a device, allowing a program interact with the device.
● It follows the interface specification defined by the OS (open, read, write, ...)4 Each OS defines its own interface
● It implements device dependent tasks 4 Each device performs specific tasks
● Usually, it contains low-level code4 I/O ports access, interrupt management,...
● Functionalities are encapsulated in a binary file
Device Driver
9/9/19
11
1.21
■ Modern kernels offer mechanisms to add data+code to the kernel without kernel recompilation● Kernel recompilation could take hours ...
■ Nowadays insertion is performed dynamically (at runtime) ● Dynamic Kernel Module Support (linux) or Plug&Play (windows)
■ Linux Module mechanism ● File(s) compiled in a specific way that contain the data+code to be
inserted into the kernel● A set of shell commands to load/unload/list modules● To be discussed in laboratory
Dynamic insertion code: Linux modules
1.22
■ Device driver for a fake device. We install the driver using a module, so we don not need to recompile the kernel.
In the Lab…MyDriver1.c
struct file_operations fops_driver_1 = {owner: THIS_MODULE,read: read_driver_1,
};
int read_driver_1 (struct file *f, char __user *buffer, size_t s, loff_t *off) {…return size;
}
static int __init driver1_init(void){…}static void __exit driver1_exit(void){…}
module_init(driver1_init);module_exit(driver1_exit);
Operations provided by the driver
In this case, only the readoperation is implemented
Operations to load/unloaddriver into/from the kernel
Module operations toload/unload
9/9/19
12
1.23
■ Internals of a Device Driver (DD)● General information about the DD: name, author, license, description, ...● Implementation of the generic functions to access devices
4 open, read, write, ...● Implementation of the specific functions to access devices
4 Device programming, access to ports, interruption management, …● Data struct with pointers to specific functions● Init function
4 Executed when the DD is installed4 Registers the DD on the System, associating to a major4 Associates the generic functions to the registered DD
● Exit function4 Unregisters the DD from the System and the associated functions
■ Example of DD: see myDriver1.c and myDriver2.c in the lab docs
Device Driver (DD) in Linux: +details
1.24
■ Steps to create and start using a new device:● Compile DD, if needed, in kernel object format (.ko)
4 Type (block/character) and IDs are especified inside the code● Install (insert) at runtime driver routines
4 insmod file_driver4 The driver is related to a given major id
● Create a logic device (filename) and link it to the physical device4 New file ßàblock | character + major & minor4 Command mknod
– mknod /dev/mydisp c major minor● Create the virtual device
– open(“/dev/mydisp”, ...);
I/O Linux Modules: +details
Phy.
Log.
Virt.
9/9/19
13
1.25
■ Operation of some logical devices: terminal, file, pipe, socket1. Terminal
● Logical level object that represents a keyboard + screen set● “Usually” processes have it as data standard input and output
2. Data file● Logical level object that represents information stored on disk. It is
interpreted as a sequence of bytes and the System deals with the offset inside the sequence.
Examples of devices (1)
1.26
■ Pipe● Logical level object that implements a temporary buffer operating as
FIFO. Data written to a pipe disappear as are read them. It is used to unidirectional data flow among processes4 Unnamed pipe, connects only processes with parenthood as it is
accessible just by inheritance4 Named pipe, allows connection to any process with enough
permissions on the device■ Socket
● Logical level object that implements a temporary buffer operating as FIFO. It is used to a two-way connection based byte streams betweenprocesses located in different hosts in a network.
● Operation similar to pipes, although much more complex implementation depending on the semantics of communication
Examples of devices(2)
9/9/19
14
1.27
LINUX I/O MANAGEMENT
1.28
■ In Linux, all devices are identified by a file (and there are several types of files)● block device● character device● directory● FIFO/pipe● symbolic link● regular file à data files or “ordinary files”● socket
■ We called “special” files those which are not data files: pipes, links, etc.
Linux file types
9/9/19
15
1.29
■ Almost all file types can be create with the syscall/command mknod. Exceptions:● Directories and soft links● Regular files. The mknod command (to be used in lab) does not create
data files■ mknod name [c | b] major minor
■ name device name■ c à character type devices, such as terminal and pseudo devices■ b à If the device is a block type device such as a tape or disk drive
which needs both cooked and raw special files, the type is b■ p à FIFO (pipe, no major/minor is needed)■ major identifying the class of the device.■ minor identifying a specific instance of a device in that class
Creating files in Linux
1.30
■ Some examples:
File types
dev name type major minor description/dev/fd0 b 2 0 floppy disk/dev/hda b 3 0 first IDE disk/dev/hda2 b 3 2 second primary partition
of first IDE disk/dev/hdb b 3 64 second IDE disk/dev/tty0 c 3 0 terminal/dev/null c 1 3 null device
9/9/19
16
1.31
KERNEL DATA STRUCTURES
1.32
■ Data structure for storing file system metadata with pointers to its data. Each inode represents an individual file. It stores:● size● type● access permissions● owner and group● file access times● number of links (number of file names pointing to the inode) ● pointers to data (multilevel indexation) à see below, at the end of this section
■ All information about a file, except file names■ Stored on disk, but there is an in-core copy for access optimization
Kernel data structures: inode
9/9/19
17
1.33
■ Each process● User Field Descriptor Table (FDT): per-process open-file table (saved in the
task_struct, ie, PCB)4 Records to which files the process is accessing 4 The file is accessed through the file descriptor, which is an index to the FT4 Each file descriptor is a virtual device4 Each field descriptor points to an entry in the Open File Table (FT)4 Fields we’ll assume: num_entry_OFT
■ Global: ● Open File Table (FT):
4 System-wide open-file management4 One entry can be shared among several processes and one process can point to
several entries.4 One entry of FT points to one entry of the Inode Table (IT)4 Fields we’ll assume: num_links, mode , offset, num_it_entry
● Inode Table (IT): 4 Active-inode table. One entry for each opened physical object. Including DD
routines. 4 Memory (in-core) copy of the disk data for optimization purposes,4 Fields we’ll assume: num_links, inode_data
Kernel data structures
1.34
Kernel data structures
...write(1,...)...
user system
process
User file descriptor
table
file table inode table
virtual logical physical
One per process(task_struct) System-wide
(shared)
System-wide(shared)
mod
e
offs
etEn
t_itEnt_ft
0 1 x
1
2
0 0
1 0
2 0
3
4
0 3 RW
- 0
1
2
3
4
refs
refs
inod
e
9/9/19
18
1.35
BASIC SYSTEM CALLS
1.36
■ Some I/O operations are time consuming, a process cannot be idle in the CPU à OS blocks the process (RUNàBLOCKED)
■ Blocking I/O: A process ask for a transfer of N bytes and waits for the call completes. Returns the number of bytes transferred.● If there are data available (even if the number of bytes is smaller than
requested) transfer is made and process returns immediately● If there are not data available the process is blocked
1. Process state changes from RUN to BLOCKED1. Process leaves CPU and is queued in a waiting processes list 2. The first process from the READY queue is moved to RUN (if
Round Robin)2. When data are available an interrupt arrives
– The ISR (Interrupt Service Routine) sends data to the CPU and enqueue the blocked process in the READY list (if Round Robin)
3. When the turn is over, the process will be put into RUN again
Blocking and non-blocking operations
9/9/19
19
1.37
■ Non-blocking I/O operation● A process ask for a transfer of N bytes. Data available at that time are
sent and immediately returns whether there are data or not.● Control returns immediately with the data have been transferred.
Blocking and non-blocking operations (1)
1.38
Syscall Descriptionopen Given a pathname, flags and mode returns an integer called the
user file descriptorread Reads N bytes from a device (identified by the file descriptor)
and saved in memorywrite Reads N bytes from memory and writes them to the device
(identified by the file descriptor)close Releases the file descriptor and and leaves it free to be reuseddup/dup2 Duplicates the file descriptor. Copies a file descriptor into the first
free slot of the user file table. It increments the count of the corresponding file table entry, which now has one more fd entry that points to it.
pipe Allows transfer of data between processes in a first-in-first-out manner
lseek Changes the offset of a data file (an entry in the File Table pointed by the fd).
BASIC I/O OPERATIONS
Syscalls open, read & write are blocking
9/9/19
20
1.39
■ So, how do you associate a name with a virtual device?■ fd = open(pathname, flags [, mode]);
● open syscall links a device (file name) to a virtual device (field descriptor)4 Is the first step that a process must take to access file data. It
checks permissions. After correct completion, process can call read/write multiple times without check permissions again.
4 open returns a file descriptor. Other file operations, such as reading, writing, seeking and closing the file use the file descriptor.
● pathname is a file name.● flags indicate the type of open. At least, one of them
4 O_RDONLY (reading)4 O_WRONLY (writing)4 O_RDWR (reading & writing)
● mode gives the file permissions if the file is being created.
Open
1.40
■ Regular files can be created with the system call open adding O_CREAT in the flags argument. The system call mknod creates special files.
■ Argument mode is mandatory● It is an OR (|) among : S_IRWXU, S_IRUSR, S_IWUSR, etc.
■ There is not a syscall for parcilly remove data in a file. However, file can be truncated to length 0 adding O_TRUNC in the flags argument.
■ Examples:4 Ex1: open(“X”,O_RDWR|O_CREAT, S_IRUSR|S_IWUSR) à If
file X did not exist it’s created, but otherwise O_CREAT has no effect.4 Ex2: open(“X”,O_RDWR|O_CREAT|O_TRUNC, S_IRWXU)à If file
X did not exist it’s created, but otherwise file X is empty now.
Open: creating
9/9/19
21
1.41
■ Open (cont): effects on the kernel data structures● The kernel allocates an entry in the file descriptor table. It will always be the first
free entry. The kernel records the index of the File Table in this entry● The kernel allocates an entry in the file table for the open file. It contains a pointer to
the in-core inode of the open file, and a field that indicates the byte offset in the file where the kernel expects the next read or write to begin.
● The kernel associates these structures in the corresponding DD (MAJOR of the symbolic name). It may happen that different entries of the FT point to the same DD
Open: data structure
...open(“name”,O_RDONLY)...
user system
process User file descriptor
tableFile table Inode table
virtual logical physical
0123
-RW
Per-process Per-system(shared)
Per-system(shared)
0012
0R
0123
W -211
0 1 x
1 1 Y
2 1 j
012
Inode of console
Inode of another dev.
Inode of file“name”
1.42
■ n = read(fd, buffer, count);● Asks for reading count bytes (characters) from the device pointed by fd
4 If there is great or equal count bytes available, it reads count bytes4 If there is less than count bytes, it reads all of them4 If there is no bytes, it’s up to the device behaviour:
– Blocking process until data available– Returns 0 immediately
4 If EOF, returns 0 immediately – The meaning of EOF it’s up to the device behaviour
● Returns n, the number of bytes actually read● The kernel updates the offset in the file table to the n; consequently,
successive reads of a file deliver the file data in sequence
ReadNum. of bytes actually read
File descriptor returned by open
Address of a data structure in the user process
Num. of bytes the user wants to read
9/9/19
22
1.43
■ n = write(fd, buffer, count);
● Asks for writing count bytes (characters) to the device pointed by fd4 If there is space on device for count bytes, it writes count (the kernel
allocates a new block if the file does not contains a block that corresponds to the byte offset to be written)
4 If there is less, it writes what fits4 If there is no space left on device, it’s up to the device behaviour:
– Blocking process until space available– Returns 0 immediately
● Returns n, the number of bytes actually written● The kernel updates the offset in the file table to the n; consequently,
successive writes of a file update the file data in sequence (when the write is complete, the kernel updates the file size entry in the inode if the file has grown larger)
WriteNum. of bytes actually written
File descriptor returned by open
Address of a data structure in the user process
Num. of bytes the user wants to read
1.44
Example: writing to a device
...printf(...)...
...write(1,...)...
...pushl $1int 0x80...
User code
C library
System library
user Operating System
syscall
012
Userfile descriptortable
Code anddata structureslogical level
device driver:write_dev()
Device programmingreturns result
I/O sussystem
9/9/19
23
1.45
Example: reading from a device
...scanf(...)...
...read(0,...)...
...pushl $0int 0x80...
User code
C library
Systemlibrary
user Operating System
syscall
012
Userfile descriptortable
Code anddata structureslogical level
device driver:llegir_disp()
Device programmingReturnsresult
I/O sussystem
RSint()
int
1.46
■ newfd = dup(fd);
● Where fd is the file descriptor being duped and newfd is the new file descriptor that references the file.
● Copies a file descriptor into the first free slot of the user file descriptor table.● Returns newfd
■ newfd = dup2(fd, newfd);
● Similar to dup, but the free slot is forced to be newfd● If newfd already refers to an open file, it is closed before duped
■ close(fd);
● Where fd is the file descriptor for the open file.● The kernel does the close operation by manipulating the file descriptor and the
corresponding file table and inode table.● If the reference count of the file table entry is greater than 1 (dup, fork) then the
kernel decrements the count and the close completes.● If the table reference count is 1, the kernels frees the entry and releases the in-
core inode (If other processes still reference the inode, the kernel decrements the inode reference count but leaves it allocated).
Dup/dup2/close
9/9/19
24
1.47
■ Pipes allow transfer of data between processes in a first-in-first-out manner and they allow also synchronization of process execution.● pipe(fd_vector); // Device for FIFO communications
4 Creates an unnamed pipe. Returns 2 file descriptors fd_vector[0] for reading, fd_vector[1] for writing the pipe (and allocates corresponding File Table entries).
4 There is no name in the VFS, so there is no any call to open.4 Only related processes, descendants of a processes that issued the
pipe call can share access to unnamed pipes● Named pipes are identical, except for the way that a process initially
accesses them● mknod("my_pipe", S_IFIFO | 0600, 0);
4 Creates a pipe, named “my_pipe”, in the VFS and, hence, processes that are not closely related can communicate.
4 Processes use the open syscall for named pipes in the same way that they open regular files.
4 The kernel allocates 2 entries in the File Table and 1 in the Inode Table.
pipe
1.48
■ Usage● Processes use the open system call for named pipes, but the pipe system call to create unnamed
pipes. ● Afterwards processes use regular system calls for files, such as read and write, and close when
manipulating pipes.● Pipes are bidirectional, but ideally each process uses it in just one direction. In this case the kernel
manages synchronization of process execution. ■ Blocking device:
4 Opening: a process that opens the named pipe for reading will sleep until another process opens the named pipe for writing, and vice versa.
4 Reading: if the pipe is empty, the process will typically sleep until another process writes data into the pipe.
4 If the count of writer processes drops to 0 and there are processes asleep waiting to read from the pipe, the kernel awakens them, and they return from their read calls without reading any data.
4 Writing: if a process writes a pipe and the pipe cannot hold all the data, the kernel marks the inode and goes to sleep waiting for data to drain from the pipe.
– If there are no processes reading from the pipe, the processes that writes the pipe receives a signal SIGPIPE à the kernel awakens the sleeping processes
4 Processes should close all non-used files descriptors, otherwise -> Blocking!■ Data structures
● 2 entries in the user File Descriptor Table (R/W)● 2 entries in the File Table (R/W)● 1 entry in the in-core Inode Table
pipe
9/9/19
25
1.49
■ lseek changes the File Table byte offset (the read-write pointer) . It allows direct access by position in data files (or even sequential devices, like tapes).● Offset is 0 after an open system call (except with APPEND flag).● Offset is increased by read and write system calls.● Offset can be modified by the user with lseek system call
■ new = lseek(fildes, offset, origin)
■ The value of the pointer depends on origin:● SEEK_SET: pointer = offset.Set the pointer to offset bytes from the beginning of the file.● SEEK_CUR: pointer += offset. Increment the current value of the pointer by offset.● SEEK_END: pointer = file_size + offset. Set the pointer to the size of the file plus offset
bytes.● offset can be negative.
lseek
1.50
■ I/O and fork● Child process inherits a copy of the parent process file descriptor table.
4 All open entries point to the same File Table entries● Parent and child sharing devices opened before fork system call● Next calls to open will be independent
I/O and concurrent execution (1)
user system
open(“f1”,O_RDONLY)fork()open(“f1”,O_WRONLY)
process 1 FDT FT IT
-
01234
01234
open(“f1”,O_RDONLY)fork()open(“f1”,O_WRONLY)
process 200
000
000
1
13
201234
6 rw2 r1 w1 w
Inherited! FT entries shared
0
New, numbers depend on order execution
0 1 x
1 3 Y
2
0111
9/9/19
26
1.51
■ I/O and exec● New process image keeps the same process’ I/O internal structures● fork+exec allows I/O redirection before process image change
I/O and concurrent execution (2)
user system
open(“f1”,O_RWDR)if (fork()==0){
close(0);open(“f2”,???)exec(...)
process 1 FDT FT IT
x
01234
01234
new code
process 20
close(0);open(“f2”,????)exec(...)
0001
2001
01234
5 rw2 rw1 r
00 1 x
1 2 Y
2
1.52
■ If a process is blocked in a I/O operation and it is interrupted by a signal there are two possible behaviours:● After the signal is handled, the kernel resumes the I/O operation (so the
process remains blocked)● After the signal is handled, operation returns error and and sets errno
to EINTR.■ Behaviour is up to
● Signal programming: 4 If flag SA_RESTART in sigaction à operation is resumed4 Otherwise à operation returns error
● The operation itself (for instance, operations on sockets, on waiting signals, etc.)
■ Example: how to protect a system call depending on behaviour:while ( (n = read(...)) == -1 && errno == EINTR );
I/O and concurrent execution (3)
9/9/19
27
1.53
■ Although system calls are uniform, devices are not■ There are system calls to modified the specific characteristics of
logical and virtual devices■ Logical device
● ioctl(fd, cmd [, ptr]);
■ Virtual device● fcntl(fd, cmd, [, args]);
■ Arguments are quite generics to offer flexibility
manipulating the underlying device parameters
1.54
■ Terminal● For each terminal the system has a buffer to keep characters typed in
order to erase some of them, if necessary, before interpret them● POSIX defines special functionalities to some characters:
4 ^H: erases a character 4 ^U: erases a line 4 ^D: EOF (end-of-file) to indicate the end of input to the shell.
● It could implement a writer buffer with cursor functionalities like “backwards n lines” or “forward in current line”
● Each controller can implement a terminal characteristics as complex as it wants, for instance, it can modified text in the middle of the line.
● Canonical/Non canonical: Character pre-processing or not before sending to the process (reading)
Characteristics of terminals
9/9/19
28
1.55
■ Terminal: operation (canonical by default)● Reading
4 Buffer keeps chars until CR (carriage return) is pressed4 If there is a sleeping process waiting chars, it receives as chars as it can4 Otherwise, chars are saved until a process ask for them4 ^D means the end of the current reading, with the chars read since then,
even if the buffer is empty. while ( (n=read(0, &car, 1)) > 0 )
write(1, &car, 1);4 This is interpreted as end-of-file (EOF), by convention.
● Writing4 It writes blocks of chars
– It could wait for the CR to be displayed on screen4 Process is not blocked
● Behaviour of devices, like blocking, can be modified by syscalls (ioctl)
Characteristics of terminals
1.56
■ Pipe● Reading
4 If there are data, a process reading from the pipe reads what it wants in a transient way
4 If there are no data, a process reading from the pipe is blocked until other process writes to the pipe.
4 If the pipe is empty and there are no writer processes (ie, all file descriptors opened for writing are now closed), reader process receives and EOF
4 Therefore, processes must close all unused file descriptors as soon as possible.
● Writing4 If there is room for the data to be written, the kernel writes the data (or
as many as possible)4 If the pipe is full, writer process goes to sleep waiting for data to drain
from the pipe4 If there are no more readers, writer process receives a signal SIGPIPE
● Behaviour of pipes, like blocking, can be modified by syscalls (fcntl)
Characteristics of pipes
9/9/19
29
1.57
■ Although network devices are I/O devices, network functionalities cannot be covered by the generic I/O operations
■ Management of network devices is performed by different mechanisms● For instance, /dev/eth0 has no inode, nor device driver
■ There are specific system calls for networks and several interfaces■ They implement different network protocols (subject XC)
Network devices
1.58
■ Socket: operation● Mechanism, introduced by the BSD Unix, to provide common methods
for interprocess communication and to allow use of network protocols.● Similar to pipes. It uses just one file descriptor for full duplex
communication.● The socket system call establishes the end point of a communications
link between two processes connected to the network. Each process executes sd = socket(format, type, protocol);
● A process can ask for connection to a remote socket and detect if someone wants to connect to a local socket. The send and recv are socket specific system calls that transmit data over a connected socket. The read and wite syscalls are also valid.
● The socket mechanism contains several system calls. They allow implement concurrent and, in general, distributed applications
● In general, sockets use a client–server architecture
Network devices
9/9/19
30
1.59
Network devices
■ Socket: Example (pseudo-code)
● Client
...
sfd = socket(...)
connect(sfd, ...)
write/read(sfd, ...)
● Server...sfd = socket(...)
bind(sfd, ...)
listen(sfd, ...)
nsfd = accept(sfd, ...)
read/write(nsfd, ...)
1.60
EXAMPLES
9/9/19
31
1.61
■ Reading from the standard input and writing to the standard output
● Note:4 Reading while there are data (n==0), that’s up to the device. The total
amount of syscall depends on the number of bytes to be read4 Processes conventionally have access to three files: its standard input (0), its
standard output (1) and its standard error(2). 4 Processes executing at a terminal typically use the terminal for these three
files.4 But each may be "redirected" independently to any logical device that
accepts the operations of reading and/or writing.4 For instance:
Byte-by-Byte access
while ((n = read(0, &c, 1)) > 0)write(1, &c, 1);
#example1 à input=terminal, output=terminal#example1 <disp1 à input=disp1, output=terminal#example1 <disp1 >disp2 à input=disp1, output=disp2
1.62
■ The same, but reading blocks if bytes (chars in this case)
● Note:4 You must write n bytes
– Process is asking for SIZE bytes, however it reads n bytes4 What about performance? How many system calls are executed?
Buffer in user space access
char buf[SIZE];...while ((n = read(0, buf, SIZE)) > 0)
write(1, buf, n);
9/9/19
32
1.63
Data communication using pipes
■ Program a process schema equivalent to the figure:
■ 2 pipes■ P1 sends to pipe1 and
receives from pipe2■ P2 the opposite
symmetrically
P2P1
1. int pipe1[2], pipe2[2],pidp1,pidp2;2. pipe(pipe1);3. pipe(pipe2);4. pidp1=fork();5. if (pidp1==0){6. close(pipe1[0]);7. close(pipe2[1]);8. p1(pipe2[0],pipe1[1]);9. exit(0);10.}11.close(pipe1[1]);12.close(pipe2[0]);13.pidp2=fork();14.if (pidp2==0){15. p2(pipe1[0],pipe2[1]);16. exit(0);17.}18.close(pipe1[0]);close(pipe2[1]);19.while(waitpid(-1,null,0)>0);
void p1(int fdin,int fdout);void p2(int fdin,int fdout);
1.64
Random access and size evaluation
■ What does this code do?
■ And this one?
fd = open(“abc.txt”, O_RDONLY);while (read(fd, &c, 1) > 0) {
write(1, &c, 1);lseek(fd, 4, SEEK_CUR);
}
fd = open(“abc.txt”, O_RDONLY);size = lseek(fd, 0, SEEK_END);printf(“%d\n”, size);
You can find this code at : ejemplo2.c
You can find this code at: ejemplo1.c
9/9/19
33
1.65
pipes and blockingint fd[2];...pipe(fd);pid = fork();if (pid == 0) { // child
while (read(0, &c, 1) > 0) {// Reads, process and send datawrite(fd[1], &c, 1);
}}else { // parent
while (read(fd[0], &c, 1) > 0) {// Receives, process and send datawrite(1, &c, 1);
}}...
■ Be careful The parent process must close fd[1] if it does not want to be blocked!
This code is available at: pipe_basic.c
1.66
■ What does this code do?
Sharing the read-writer pointer
...fd = open(“fitxer.txt”, O_RDONLY);pid = fork();while ((n = read(fd, &car, 1)) > 0 )
if (car == ‘A’) numA++;sprintf(str, “El número d’As és %d\n”, numA);write(1, str, strlen(str));...
This code is available at: exemple1.c
9/9/19
34
1.67
■ What does this code do?
Non shared read-write pointer
...pid = fork();fd = open(“fitxer.txt”, O_RDONLY);while ((n = read(fd, &car, 1)) > 0 )
if (car == ‘A’) numA++;sprintf(str, “El número d’As és %d\n”, numA);write(1, str, strlen(str));...
This code is available at: exemple2.c
1.68
■ What does this code do?
Redirection of standard input and output
...pid = fork();if ( pid == 0 ) {
close(0);fd1 = open(“/dev/disp1”, O_RDONLY);close(1);fd2 = open(“/dev/disp2”, O_WRONLY);execv(“programa”, “programa”, (char *)NULL);
}...
9/9/19
35
1.69
Redirection and pipes
...pipe(fd);pid1 = fork();if ( pid1 != 0 ) { // parent
pid2 = fork();if ( pid2 != 0 ) { // parent
close(fd[0]); close(fd[1]);while (1);
}else { // child 2
close(0); dup(fd[0]);close(fd[0]); close(fd[1]);execlp(“programa2”, “programa2”, NULL);
}}else { // child 1
close(1); dup(fd[1]);close(fd[0]); close(fd[1]);execlp(“programa1”, “programa1”, NULL);
}
1.70
■ Write a fragment of code that creates two processes P1 and P2, connect them by two pipes using the standard I/O file descriptors: in the first one P1writes and P2 reads, in the second one P2 writes and P1 reads
■ Write a fragment of code that creates a sequence of N processes in chain: each process Pi creates only one child Pi+1, until PN. Each process communicates in chain with its parent and its child via a pipe in the standard I/O file descriptors, so that what the first process wrote arrives to the last process
Classroom exercises (1)
P2P1
P2P1 P3 PN...
9/9/19
36
1.71
■ Write a fragment of code that creates N processes in sequence: the initial process creates all children, from P1 until PN. Each process communicates with the parent process via a pipe; the parent process must be able to write to all the pipes (using file descriptors 3..N+2) and children must read from its standard input.
Classroom exercises (2)
P2P1 P3 PN...
Pinit
1.72
FILE SYSTEM
9/9/19
37
1.73
■ Duties of the file system regarding file management and data storage● Organize files in the systemà name space and directories schema● Guarantee correct access to files (access permissions)● Used/free space management (allocation/releasing) for data files● Search/storage data of files
■ At all events, each file (of any kind) has a name and it must be stored and managed. File names are organized in directories.
File system tasks
1.74
■ Directory: logical structure that organizes files.■ It is a kind of file (type directory) OS managed (users cannot open it or
handled). ■ It allows linking file name and the file attributes
● File attributes4 File type: directory(d)/block(b)/character(c)/pipe(p)/link(l),socket(s),
regular(-)4 size4 owner4 permissions4 file access times4 ...
● Table of contents for the disk addresses of data in a file (in case of data files)
Name space: directories
In L
inux
, al
l thi
s st
uff
is in
the
inod
e
9/9/19
38
1.75
■ Directories are organized in a hierarchical way (graph)■ Directories let users to classify their data■ The file system organizes the storage devices (each one with its own
directory schema) in a single name space with an unique entry point■ The entry point is the root directory, that is, “/”■ Any directory has, at least, two (special) files (also directories)
● . Dot: link to the current directory
● .. Dot-dot: link to the parent directory
Linux: user view
1.76
/
. .. home
. .. usr1
. .. F1 F2
usr2
. ..
Appl
. .. A B
■ Each file can be searched in two ways:● Absolute path name: the path name starts with the slash character● Relative path name: starts from the current directory of the process
Linux: user view
If you are here, you can search for f2 using:Relative path name: f2Absolute path name: /home/usr1/F2
9/9/19
39
1.77
Linux: file names■ File name is not among the file attributes (inode). Linux allows more than on file name with
the same inode number.■ There two types of links between file name and inode.
● Hard-link:4 File name points to the inode that contains file attributes and data.4 It’s the most common4 File attributes include a reference counter (how many file names for this inode).
– Be careful: this ref. counter is different from the one stored in the in-core inode table!
4 Command line: ln origin destination4 System call: link(origin, destination);
● Soft-link4 File name points to a inode that contains the path name of the destination file4 It is not a regular file (l)4 Command line: ln -s origin destination 4 System call: symlink(origin, destination);
1.78
■ The existence of the two types of links influences the directory structure (graph) ● No cycles are allowed with hard links● Cycles are allowed with soft links
Directory hierarchy implementation
acyclic-graphFS checks that no cycles are created
cyclic-graphFS checks for infinite loops
9/9/19
40
1.79
Problems with directories in graph
■ Backups ● Do not backup a file twice
■ Deleting files● Soft links
4 The kernel does not check for soft links to a file. In access time it will detect that the destination if file does not exist
● Hard links4 Inode is deleted when the reference count is zero
1.80
Directories internal view■ A directory is a file whose data is a sequence of entries, each consisting of
an inode number and the name of a file contained in the directory■ For instance:
Nom inodo. 2.. 2home 3Appl 7
/(2)
. .. home(3)
. .. usr1(3)
. .. F1(4) F2(5)
usr2(6)
. ..
Appl(7)
. .. A(8) B(9)
Nom inodo. 2.. 3F1 4F2 5
Directory / Directory /home/usr1Special case
9/9/19
41
1.81
■ The FS lets assign different permissions to files● FS defines levels of access and operations that are allowed
■ Linux:● Levels of access: owner, group, world● Operations: Read (r) , Write (w), Execute (x) à Be careful: this is not
the flag access mode in the open() syscall! ● Access permissions can be changed with a number in octal. Some typic
values
■ See laboratory documentation
File access permissions
R W X Valor numèric1 1 1 71 1 0 61 0 0 4
1.82
System calls: name space and permissions
■ There are more system calls that allow processes to change :● Permissions● Characteristics● etc
Service System call
Create / remove link to file/soft-link
link / unlink/symlink
Changing file permissions
chmod
Changing owner and group
chown / chgrp
query the status of files stat, lstat, fstat
9/9/19
42
1.83
■ A storage device is divided in parts named sectors■ The OS allocation unit is the disk block (1 block corresponds to 1 or more
sectors)
■ Partition, volume or file system● a large sequential array of logical blocks, with an unique identifier
(logical device). They are manged by the OS as a independent logical entity. 4 C:, D: (Windows); /dev/hda1, /dev/hda2 (Linux); /dev/disk1s1 (MacOS X)
● Each partition has its own file system independent of the rest of partitions
Disk work units
1.84
Accessing partitions■ A file system is accessible after execution of the mount system call or shell
command (only root user can do that)■ Linux command line:
4 # mount –t ext2 /dev/hda1 /home4 # umount /home
■ Where “ext2” is the file system type, “dev/hda1” is the name of the device partition and “home” is the mount point, i.e., the location within the file structure where the file system is going to be attached.
/
bin
etc
usrmnthome
cdromdvd
usercdrom
user1user2user3
/
album1album2album3
/
/
bin
etc
usrmnthome
cdromdvd
user1user2user3 album1
album2album3
9/9/19
43
1.85
■ For each file, OS must know where are located its blocks. ● Indexed allocation: pointers to data blocks assigned to a file● Inode contains the indexes. How many?
4 multilevel index■ Indexes’ blocks (B = 1KB, @ = 32b)
● 10 direct blocks4 ( 10 blocks = 10/KB)
● 1 indirect block4 (256 KB)
● 1 double indirect block4 (64 MB)
● 1 triple indirect block4 (16GB)
● Classical AT&T System V UNIX approach (1983).
Managing used space
1.86
■ Managing free space: ● Lists of free disk blocks and free inodes● When the kernel wants to allocate a block from a file system it
allocates the next available block in the list. Similar for inodes
Allocation of blocks
9/9/19
44
1.87
■ Persistent metadata: stored on disk● inodes and block list of the file● directories● a list of free blocks available on the file system● a list of free inodes in the file system,● … and any data that describes the filesystem such us: root inode, block
size, etc.)■ Superblok: data block per partition that contains file system (redundant to
be fault tolerant)■ Superblock is in memory in order to reduce disk I/O access. The kernel
periodically writes the super block to disk if it had been modified so that it is consistent with the data in the file system.
Metadata
1.88
■ Memory zone to save the last read inodes■ Memory zone to save the last read directories■ Buffer cache: memory zone to save the last read blocks■ Superblock: it is also in memory (in-core superblock)
Memory and metadata
9/9/19
45
1.89
RELATIONSHIP BETWEEN SYSTEM CALLS AND DATA STRUCTURES
1.90
Open■ The kernel searches the file system for the file name parameter and points to
one entry in the the in-core inode table..■ Which inode? Read the directory entry. Two cases:
4 Directory is in the cache. Just access the directory in cache.4 Directory is on disk. Read it and keep it in memory
– Which blocks it must read? Check inode– Which inode? Check directory (repeat the algorithm)
■ If the inode is found● The kernel checks permissions for opening the file and returns errno if
applicable. ● If it is a soft-link, it reads the inode path and searches for the file name
(repeat algorithm)● The kernel allocates an entry in a private table in the process (user File
Descriptor Table) and allocates an entry in the file table for the open file (File Table).
■ If the inode is not found: ● Error, except O_CREAT.
■ At all events, open does not access to any data block of the file requested
9/9/19
46
1.91
■ If a process calls open to create a file (O_CREAT)● Get inode for the file name
4 Update list of free inodes in the Superblock● create new directory entry in the parent directory:
4 Directory data block: include new file name and newly assigned inode number;
4 Directory inode: update size
Open (cont.)
1.92
Read
■ Get File Table entry from the user file descriptor table■ Get inode from the File Table (lock inode)■ Get the offset from the File Table ■ While count not satisfied
● EOF?● calculate offset into block,
4 By dividing the value of the pointer by the block size, the kernel obtains the index of the first block to read
■ If the blocks were not in memory, the kernel reads them from disk (and keep them in the buffer cache)
9/9/19
47
1.93
■ Similar to that for reading a regular file■ In this case, if the file does not contain a block that corresponds to the byte
offset to be written, the kernel allocates a new block● Superblock:
4 Update the list of free-block● Inode:
4 Assigns the block number to the correct position in the inode’s table of contents
4 Update inode file size
Write
1.94
■ The kernel performs the close operation by manipulating the file descriptor and the corresponding file table and inode table entries.
■ FDT: the user file descriptor table entry is empty. ■ File Table
● Decrements the file reference count. If zero, frees the entry and releases the in-core inode.
■ Inode Table● Decrements the inode reference count. If zero, the inode is free for
reallocation■ Inode:
● Updates time stamps
Close
