Input and Output

TopicsTopics I/O hardware Unix file abstraction Robust I/O File sharing

CS 105“Tour of the Black Holes of Computing”

– 2 – CS 105

I/O: A Typical Hardware System

mainmemory

I/O bridgebus interface

ALU

register file

CPU chip

system bus memory bus

disk controller

graphicsadapter

USBcontroller

mousekeyboard monitordisk

I/O bus Expansion slots forother devices suchas network adapters.

– 3 – CS 105

Abstracting I/OLow level requires complex device commandsLow level requires complex device commands

Vary from device to device Device models can be very different

Tape: read or write sequentially, or rewindDisk: “random” access at block levelTerminal: sequential, no rewind, must echo and allow editingVideo: write-only, with 2-dimensional structure

Operating system should hide these differencesOperating system should hide these differences “Read” and “write” should work regardless of device Sometimes impossible to generalize (e.g., video) Still need access to full power of hardware

– 4 – CS 105

Unix FilesA Unix A Unix filefile is a sequence of is a sequence of mm bytes: bytes:

B0, B1, .... , Bk , .... , Bm-1

All I/O devices are represented as files:All I/O devices are represented as files: /dev/sda2 (/usr disk partition) /dev/tty2 (terminal)

Even the kernel is represented as files:Even the kernel is represented as files: /dev/kmem (kernel memory image) /proc (kernel data structures)

– 5 – CS 105

Unix File TypesRegular fileRegular file: binary or text. Unix does not know the : binary or text. Unix does not know the

difference!difference!DirectoryDirectory: contains the names and locations of other : contains the names and locations of other

filesfilesCharacter special fileCharacter special file: keyboard and network, for : keyboard and network, for

exampleexampleBlock special fileBlock special file: like disks : like disks FIFOFIFO (named pipe): used for interprocess (named pipe): used for interprocess

comunicationcomunicationSocketSocket: used for network communication between : used for network communication between

processesprocesses

– 6 – CS 105

Unix I/OThe elegant mapping of files to devices allows kernel to The elegant mapping of files to devices allows kernel to

export a simple interface called Unix I/Oexport a simple interface called Unix I/O

Key Unix ideaKey Unix idea: All input and output is handled in a : All input and output is handled in a consistent and uniform wayconsistent and uniform way

Basic Unix I/O operations (system calls): Basic Unix I/O operations (system calls): Opening and closing files: open()and close() Changing the current file position (seek): llseek (not

discussed) Reading and writing a file: read() and write()

– 7 – CS 105

Opening Files

Opening a file tells kernel you are getting ready to access itOpening a file tells kernel you are getting ready to access itReturns small identifying integer Returns small identifying integer file descriptorfile descriptor

fd == -1 indicates that an error occurred; errno has reason strerror converts to English (Note: use strerror_r for thread safety)

Each process created by a Unix shell begins life with three open files Each process created by a Unix shell begins life with three open files (normally connected to terminal):(normally connected to terminal): 0: standard input 1: standard output 2: standard error

int fd; /* file descriptor */

if ((fd = open(“/etc/hosts”, O_RDONLY)) == -1) { fprintf(stderr, “Couldn’t open /etc/hosts: %s”, strerror(errno)); exit(1);}

– 8 – CS 105

Closing Files

Closing a file tells kernel that you’re finished with itClosing a file tells kernel that you’re finished with it

Closing an already closed file is recipe for disaster in Closing an already closed file is recipe for disaster in threaded programs (more on this later)threaded programs (more on this later)

Some error reports are delayed until close!Some error reports are delayed until close!

Moral: Always check return codes, even for seemingly Moral: Always check return codes, even for seemingly benign functions such as benign functions such as close()close()

perrorperror is simplified is simplified strerror/fprintfstrerror/fprintf; see man page; see man page

int fd; /* file descriptor */int retval; /* return value */

if ((retval = close(fd)) == -1) { perror(“close”); exit(1);}

– 9 – CS 105

Reading Files

Reading a file copies bytes from current file position to memory, Reading a file copies bytes from current file position to memory, then updates file positionthen updates file position

Returns number of bytes read from file Returns number of bytes read from file fdfd into into bufbuf nbytes == -1 indicates error occurred; 0 indicates end of file (EOF) short counts (nbytes < sizeof(buf) ) are possible and are not

errors!

char buf[4096];int fd; /* file descriptor */unsigned int nbytes; /* number of bytes read */

/* Open file fd ... *//* Then read up to 4096 bytes from file fd */if ((nbytes = read(fd, buf, sizeof(buf))) == -1) { perror(“read”); exit(1);}

– 10 – CS 105

Writing Files

Writing a file copies bytes from memory to current file position, Writing a file copies bytes from memory to current file position, then updates current file positionthen updates current file position

Returns number of bytes written from Returns number of bytes written from bufbuf to file to file fdfd nbytes == -1 indicates that an error occurred As with reads, short counts are possible and are not errors!

Here, transfers up to 4096 bytes from address Here, transfers up to 4096 bytes from address bufbuf to file to file fdfd

char buf[4096];int fd; /* file descriptor */unsigned int nbytes; /* number of bytes read */

/* Open the file fd ... *//* Then write up to 4096 bytes from buf to file fd */if ((nbytes = write(fd, buf, sizeof(buf)) == -1) { perror(“write”); exit(1);}

– 11 – CS 105

Simple Example

Copying standard input to standard output one byte Copying standard input to standard output one byte at a timeat a time

Note the use of error-handling wrappers for read Note the use of error-handling wrappers for read and write (Appendix B in text)and write (Appendix B in text)

#include "csapp.h"

int main(void) { char c;

while(Read(STDIN_FILENO, &c, 1) == 1) Write(STDOUT_FILENO, &c, 1);

exit(0);}

– 12 – CS 105

Dealing with Short CountsShort counts can occur in these situations:Short counts can occur in these situations:

Encountering (end-of-file) EOF on reads Reading text lines from a terminal Reading and writing network sockets or Unix pipes

Short counts never occur in these situations:Short counts never occur in these situations: Reading from disk files, except for EOF Writing to disk files

How should you deal with short counts in your code?How should you deal with short counts in your code? Use the RIO (Robust I/O) package from your textbook’s csapp.c file (Appendix B)

(But note that it handles EOF wrong on terminal) Use C stdio or C++ streams Write your code very, very carefully Ignore the problem and accept that your code is fragile

– 13 – CS 105

“Foolproof” I/O

Low-level I/O is difficult because of short counts and Low-level I/O is difficult because of short counts and other possible errorsother possible errors

Textbook provides RIO package, a (fairly) good Textbook provides RIO package, a (fairly) good example of how to encapsulate low-level I/Oexample of how to encapsulate low-level I/O

RIO is set of wrappers that provide efficient and RIO is set of wrappers that provide efficient and robust I/O in applications (e.g., network programs) robust I/O in applications (e.g., network programs) that are subject to short counts.that are subject to short counts.

Download from Download from csapp.cs.cmu.edu/public/ics/code/src/csapp.c csapp.cs.cmu.edu/public/ics/code/src/csapp.c csapp.cs.cmu.edu/public/ics/code/include/csapp.hcsapp.cs.cmu.edu/public/ics/code/include/csapp.h

– 14 – CS 105

Implementation of rio_readn/* * rio_readn - robustly read n bytes (unbuffered) */ssize_t rio_readn(int fd, void *usrbuf, size_t n) { size_t nleft = n; ssize_t nread; char *bufp = usrbuf;

while (nleft > 0) {if ((nread = read(fd, bufp, nleft)) == -1) { if (errno == EINTR) /* interrupted by signal

handler return */nread = 0; /* so call read() again */

elsereturn -1; /* errno set by read() */

} else if (nread == 0) break; /* EOF */nleft -= nread;bufp += nread;

} return (n - nleft); /* return >= 0 */}

– 15 – CS 105

Unbuffered I/ORIO provides buffered and unbuffered routinesRIO provides buffered and unbuffered routines

Unbuffered:Unbuffered: Especially useful for transferring data on network sockets Same interface as Unix read and write rio_readn returns short count only if it encounters EOF

Usually incorrect if reading from terminal rio_writen never returns a short count Calls to rio_readn and rio_writen can be interleaved

arbitrarily on the same descriptor Small unbuffered I/Os are inefficient

– 16 – CS 105

Buffered InputBuffered:Buffered:

Efficiently read text lines and binary data from file partially cached in an internal memory buffer

rio_readlineb reads text line of up to maxlen bytes from file fd and stores it in usrbuf. Especially useful for reading lines from network sockets

rio_readnb reads up to n bytes from file fd Calls to rio_readlineb and rio_readnb can be

interleaved arbitrarily on same descriptorWarning: Don’t interleave with calls to rio_readn with

calls to *b versions

– 17 – CS 105

Buffered Example

Copying the lines of a text file from standard input Copying the lines of a text file from standard input to standard outputto standard output

#include "csapp.h"

int main(int argc, char **argv) { int n; rio_t rio; char buf[MAXLINE];

Rio_readinitb(&rio, STDIN_FILENO); while((n = Rio_readlineb(&rio, buf, MAXLINE)) != 0)

Rio_writen(STDOUT_FILENO, buf, n); exit(0);}

– 18 – CS 105

I/O ChoicesUnix I/OUnix I/O

• Most general and basic; others are implemented using it• Unbuffered; efficient input requires buffering• Tricky and error-prone; short counts, for example

Standard I/OStandard I/O• Buffered; tricky to use on network sockets• Potential interactions with other I/O on streams and sockets• Not all info is available (see later slide on metadata)

RIORIOC++ streamsC++ streamsRoll your ownRoll your own

– 19 – CS 105

I/O Choices, continuedUnix I/OUnix I/OStandard I/OStandard I/ORIORIO

• Buffered and unbuffered• Nicely packaged• Author’s choice for sockets and pipes

• But buffered version has problems dealing with EOF on terminals

• Non-standard, but built on Stevens’s work

C++ streamsC++ streams• Standard (sort of)• Very complex

Roll your ownRoll your own• Time consuming• Error-prone

Unix Bible: W. Richard Stevens, Unix Bible: W. Richard Stevens, Advanced Programming in the Advanced Programming in the Unix Environment,Unix Environment, Addison Addison Wesley, 1993.Wesley, 1993.

– 20 – CS 105

How the Unix KernelRepresents Open FilesTwo descriptors referencing two distinct open filesTwo descriptors referencing two distinct open files

Descriptor 1 (stdout) points to terminal, and descriptor 4 points to Descriptor 1 (stdout) points to terminal, and descriptor 4 points to open disk fileopen disk file

fd 0fd 1fd 2fd 3fd 4

Descriptor table[one table per process]

Open file table [shared by all processes]

v-node table[shared by all processes]

File posrefcnt=1

...

File posrefcnt=1

...

stderrstdoutstdin File access

...

File sizeFile type

File access

...

File sizeFile type

File A (terminal)

File B (disk)

Info in stat struct

– 21 – CS 105

File SharingTwo distinct descriptors sharing the same disk file through Two distinct descriptors sharing the same disk file through

two distinct open file table entriestwo distinct open file table entries E.g., Calling open twice with the same filename argument

fd 0fd 1fd 2fd 3fd 4

Descriptor table(one table

per process)

Open file table (shared by

all processes)

v-node table(shared by

all processes)

File posrefcnt=1

...

File posrefcnt=1

...

File access

...

File sizeFile type

File A

File B

– 22 – CS 105

How Processes Share Files

A child process inherits its parent’s open files. Here is the A child process inherits its parent’s open files. Here is the situation immediately after a situation immediately after a forkfork

fd 0fd 1fd 2fd 3fd 4

Descriptor tables

Open file table (shared by

all processes)

v-node table(shared by

all processes)

File posrefcnt=2

...

File posrefcnt=2

...

Parent's table

fd 0fd 1fd 2fd 3fd 4

Child's table

File access

...

File sizeFile type

File access

...

File sizeFile type

File A

File B

– 23 – CS 105

I/O RedirectionQuestion: How does a shell implement I/O redirection?Question: How does a shell implement I/O redirection?

unix> ls > foo.txt

Answer: By calling the Answer: By calling the dup2(oldfd, newfd)dup2(oldfd, newfd) function function Copies (per-process) descriptor table entry oldfd to entry newfd

a

b

fd 0fd 1fd 2fd 3fd 4

Descriptor tablebefore dup2(4,1)

b

b

fd 0fd 1fd 2fd 3fd 4

Descriptor tableafter dup2(4,1)

– 24 – CS 105

File MetadataMetadataMetadata is data about data, in this case file data. is data about data, in this case file data.

Maintained by kernel, accessed by users with the Maintained by kernel, accessed by users with the stat stat and and fstatfstat functions. functions.

/* Metadata returned by the stat and fstat functions */struct stat { dev_t st_dev; /* device */ ino_t st_ino; /* inode */ mode_t st_mode; /* protection and file type */ nlink_t st_nlink; /* number of hard links */ uid_t st_uid; /* user ID of owner */ gid_t st_gid; /* group ID of owner */ dev_t st_rdev; /* device type (if inode device) */ off_t st_size; /* total size, in bytes */ unsigned long st_blksize; /* blocksize for filesystem I/O */ unsigned long st_blocks; /* number of blocks allocated */ time_t st_atime; /* time of last access */ time_t st_mtime; /* time of last modification */ time_t st_ctime; /* time of last change */};

– 25 – CS 105

Summary: Goals of Unix I/OUniform viewUniform view

User doesn’t see actual devices Devices and files look alike (to extent possible)

Uniform drivers across devicesUniform drivers across devices ATA disk looks same as IDE, EIDE, SCSI, … Tape looks pretty much like disk

Support for many kinds of I/O objectsSupport for many kinds of I/O objects Regular files Directories Pipes and sockets Devices Even processes and kernel data