+ All Categories
Home > Documents > Hung-Wei TsengFile system Device independent I/O interface (e.g. ioctl) fread/fwrite/ fopen/fclose...

Hung-Wei TsengFile system Device independent I/O interface (e.g. ioctl) fread/fwrite/ fopen/fclose...

Date post: 03-Aug-2020
Category:
Upload: others
View: 3 times
Download: 0 times
Share this document with a friend
36
File system basics Hung-Wei Tseng
Transcript
Page 1: Hung-Wei TsengFile system Device independent I/O interface (e.g. ioctl) fread/fwrite/ fopen/fclose open/close How your application reaches H.D.D. !18 Device Driver Device Driver Device

File system basicsHung-Wei Tseng

Page 2: Hung-Wei TsengFile system Device independent I/O interface (e.g. ioctl) fread/fwrite/ fopen/fclose open/close How your application reaches H.D.D. !18 Device Driver Device Driver Device

Recap: von Neumman Architecture

!2

Processor

MemoryStorage

Program0f00bb27509cbd23 00005d24 0000bd24 2ca422a0 130020e4 00003d24 2ca4e2b3Ins

tructi

ons 00c2e800

00000008 00c2f000 00000008 00c2f800 00000008 00c30000 00000008

Data

0f00bb27509cbd23 00005d24 0000bd24 2ca422a0 130020e4 00003d24 2ca4e2b3Ins

tructi

ons 00c2e800

00000008 00c2f000 00000008 00c2f800 00000008 00c30000 00000008

Data

0f00bb27

00c2e800

509cbd23

By loading different programs into memory, your computer can perform different functions

Page 3: Hung-Wei TsengFile system Device independent I/O interface (e.g. ioctl) fread/fwrite/ fopen/fclose open/close How your application reaches H.D.D. !18 Device Driver Device Driver Device

• How our systems interact with I/O • The basics of storage devices • File

!3

Outline

Page 4: Hung-Wei TsengFile system Device independent I/O interface (e.g. ioctl) fread/fwrite/ fopen/fclose open/close How your application reaches H.D.D. !18 Device Driver Device Driver Device

The computer is now like a small network

!4

SATA SSD

HDD

Wireless NIC

NIC

Processor

DRAM

processor-memory bus

GPU Accelerator

NVMe SSDFPGA/ASIC

Physical main memory is not directly linking to the system interconnect

Page 5: Hung-Wei TsengFile system Device independent I/O interface (e.g. ioctl) fread/fwrite/ fopen/fclose open/close How your application reaches H.D.D. !18 Device Driver Device Driver Device

• Registers • Command: receiving commands from host • Status: tell the host the status of the device • Data: the location of exchanging data

• Microcontroller • Memory • ASICs

!5

What’s in each device?

Registers Microcontroller

MemoryASICs

ASIC (e.g. NAND)DRAM

Controller + Registers

Page 6: Hung-Wei TsengFile system Device independent I/O interface (e.g. ioctl) fread/fwrite/ fopen/fclose open/close How your application reaches H.D.D. !18 Device Driver Device Driver Device

• The device signals the processor only when the device requires the processor/OS handle some tasks/data

• The processor only signals the device when necessary

!6

Interrupt

System Interconnect

CPU System Memory

(3)

(4) (1)

(2)

Registers Microcontroller

MemoryASICs

Page 7: Hung-Wei TsengFile system Device independent I/O interface (e.g. ioctl) fread/fwrite/ fopen/fclose open/close How your application reaches H.D.D. !18 Device Driver Device Driver Device

System Interconnect

• The processor/OS constantly asks if the device (e.g. examine the status register of the device) is ready to or requires the processor/OS handle some tasks/data

• The OS/processor executes corresponding handler if the device can handle demand tasks/data or has tasks/data ready

!7

Polling

(2) (3)

CPU System Memory

Registers Microcontroller

MemoryASICs

(1) (4)

Page 8: Hung-Wei TsengFile system Device independent I/O interface (e.g. ioctl) fread/fwrite/ fopen/fclose open/close How your application reaches H.D.D. !18 Device Driver Device Driver Device

How your application interact with peripherals

!8

Device Driver Device Driver Device Driver

Device Controller Device Controller Device Controller

Device #1 Device #2 Device #3 Device #4

User

Kernel

Hardware

Applications

I/O libraries

Applications with Direct I/O

Bufferdata

Page 9: Hung-Wei TsengFile system Device independent I/O interface (e.g. ioctl) fread/fwrite/ fopen/fclose open/close How your application reaches H.D.D. !18 Device Driver Device Driver Device

Case study: interacting with hard disk drives

!9

Page 10: Hung-Wei TsengFile system Device independent I/O interface (e.g. ioctl) fread/fwrite/ fopen/fclose open/close How your application reaches H.D.D. !18 Device Driver Device Driver Device

• Position the head to proper track (seek time)

• Rotate to desired sector.(rotational delay)

• Read or write data from/to disk to in the unit of sectors (e.g. 512B)

• Takes at least 5ms for each access

!10

Hard Disk Drivetracksector

cylinder

Each sector is identified, locate by an “block address”

head

Page 11: Hung-Wei TsengFile system Device independent I/O interface (e.g. ioctl) fread/fwrite/ fopen/fclose open/close How your application reaches H.D.D. !18 Device Driver Device Driver Device

• SATA II (300MB/s in theory), 7200 R.P.M., seek time around 8 ms. Assume the controller overhead is 0.2ms. What’s the latency and bandwidth of accessing a 512B sector?

!11

Seagate Barracuda 12

Latency = seek time + rotational delay + transfer time + controller overhead

Bandwidth = volume_of_data over period_of_time

8 ms + 12 × 1

720060

+0.5

1024300 +0.2 ms

= 8 ms + 4.17 ms + 0.00167 us + 0.2 ms = 12.36 ms

= 0.5KB12.36ms = 40.45KB/sec

Page 12: Hung-Wei TsengFile system Device independent I/O interface (e.g. ioctl) fread/fwrite/ fopen/fclose open/close How your application reaches H.D.D. !18 Device Driver Device Driver Device

• SATA II (300MB/s in theory), 7200 R.P.M., seek time around 8 ms. Assume the controller overhead is 0.2ms. What’s the latency of accessing a consecutive 4MB data?

!12

Seagate Barracuda 12

Trading latencies with bandwidth

Latency = seek time + rotational delay + transfer time + controller overhead

Bandwidth = volume_of_data over period_of_time

8 ms + 12 × 1

720060

+ 4300 +0.2 ms

= 8 ms + 4.17 ms + 13.33 ms + 0.2 ms = 25.69 ms

= 4MB25.69ms = 155.7 MB/sec

Page 13: Hung-Wei TsengFile system Device independent I/O interface (e.g. ioctl) fread/fwrite/ fopen/fclose open/close How your application reaches H.D.D. !18 Device Driver Device Driver Device

Numbering the disk space with block addresses

!13

tracksector

cylinder

0 78 15

16 2324 3132 3940 4748 5556 63

Disk blocks

Page 14: Hung-Wei TsengFile system Device independent I/O interface (e.g. ioctl) fread/fwrite/ fopen/fclose open/close How your application reaches H.D.D. !18 Device Driver Device Driver Device

How your application interact with peripherals

!14

Device Driver Device Driver Device Driver

Device Controller Device Controller Device Controller

Device #1 Device #2 Device #3 Device #4

User

Kernel

Hardware

Applications

I/O libraries

Applications with Direct I/O

Bufferdata

The application needs to be tightly coupled with the underlying device — Not generic Not portable

read/write — 0, 512, 4096, … (block address)

Page 15: Hung-Wei TsengFile system Device independent I/O interface (e.g. ioctl) fread/fwrite/ fopen/fclose open/close How your application reaches H.D.D. !18 Device Driver Device Driver Device

–David Wheeler

All problems in computer science can be solved by another level of indirection

!15

Page 16: Hung-Wei TsengFile system Device independent I/O interface (e.g. ioctl) fread/fwrite/ fopen/fclose open/close How your application reaches H.D.D. !18 Device Driver Device Driver Device

The file & file system abstraction

!16

Page 17: Hung-Wei TsengFile system Device independent I/O interface (e.g. ioctl) fread/fwrite/ fopen/fclose open/close How your application reaches H.D.D. !18 Device Driver Device Driver Device

How your application interact with peripherals

!17

Device Driver Device Driver Device Driver

Device Controller Device Controller Device Controller

Device #1 Device #2 Device #3 Device #4

User

Kernel

Hardware

Applications

I/O libraries

Applications with Direct I/O

Bufferdata

File system

Device independent I/O interface (e.g. ioctl)

fread/fwrite/fopen/fclose open/close

Page 18: Hung-Wei TsengFile system Device independent I/O interface (e.g. ioctl) fread/fwrite/ fopen/fclose open/close How your application reaches H.D.D. !18 Device Driver Device Driver Device

How your application reaches H.D.D.

!18

Device Driver Device Driver Device Driver

Device Controller Device Controller Device Controller

Device #1 Device #2 Device #3 Device #4

User

Kernel

Hardware

Applications

I/O libraries Bufferdata

File system

Device independent I/O interface (e.g. ioctl)

fread/fwrite — input.bin/output.bin

fread/fwrite — input.bin/output.bin

Bufferdata

read/write — 0, 512, 4096, … (block address)

read/write — block addresses

read/write — block addresses

The application only needs to interact with files!

Page 19: Hung-Wei TsengFile system Device independent I/O interface (e.g. ioctl) fread/fwrite/ fopen/fclose open/close How your application reaches H.D.D. !18 Device Driver Device Driver Device

What we’ve learned in the past…

!20

Page 20: Hung-Wei TsengFile system Device independent I/O interface (e.g. ioctl) fread/fwrite/ fopen/fclose open/close How your application reaches H.D.D. !18 Device Driver Device Driver Device

• Namespace has tree-like structure • Root directory (/) with subdirectories, each containing its own

subdirectories • Links break the tree analogy

!22

Hierarchical File System Structure

/

usr home var

local bin hungwei tyler spool logsrc

tylervim

Page 21: Hung-Wei TsengFile system Device independent I/O interface (e.g. ioctl) fread/fwrite/ fopen/fclose open/close How your application reaches H.D.D. !18 Device Driver Device Driver Device

• The “/“ on storage device A will become /backup now!

!23

Mount

Storage Device A

/

usr home var

local bin hungwei tyler spool logsrc

tylervim

Storage Device B/

usr home var backup

Page 22: Hung-Wei TsengFile system Device independent I/O interface (e.g. ioctl) fread/fwrite/ fopen/fclose open/close How your application reaches H.D.D. !18 Device Driver Device Driver Device

int fd, nr, nw; void *in_buff; in_buff = malloc(BUFF_SIZE);

fd1 = open(“infile.txt”, O_RDONLY); fd2 = open(“outfile.txt”, O_RDWR | O_CREAT); nr = read(fd1, in_buff, BUFF_SIZE); nw = write(fd2, in_buff, BUFF_SIZE); lseek(fd1, -8, SEEK_END); nr = read(fd1, in_buff, 8); // read last 8 bytes // more fancy stuff here… close(fd1); close(fd2);

!24

How you access files in C

Page 23: Hung-Wei TsengFile system Device independent I/O interface (e.g. ioctl) fread/fwrite/ fopen/fclose open/close How your application reaches H.D.D. !18 Device Driver Device Driver Device

Kernel

File System

open

!25

infile.txt

fd PIDs Location0 8,12123

fd = open(“infile.txt”); 22

file descriptor table

1

Page 24: Hung-Wei TsengFile system Device independent I/O interface (e.g. ioctl) fread/fwrite/ fopen/fclose open/close How your application reaches H.D.D. !18 Device Driver Device Driver Device

Kernel

File System

read

!26

infile.txt

fd PIDs Location0 8,12123

read(fd, buff, n); 22

file descriptor table

1

buff:

Page 25: Hung-Wei TsengFile system Device independent I/O interface (e.g. ioctl) fread/fwrite/ fopen/fclose open/close How your application reaches H.D.D. !18 Device Driver Device Driver Device

The design of a file system

!28

Page 26: Hung-Wei TsengFile system Device independent I/O interface (e.g. ioctl) fread/fwrite/ fopen/fclose open/close How your application reaches H.D.D. !18 Device Driver Device Driver Device

Recap: Numbering the disk space with block addresses

!29

tracksector

cylinder

0 78 15

16 2324 3132 3940 4748 5556 63

Disk blocks

Page 27: Hung-Wei TsengFile system Device independent I/O interface (e.g. ioctl) fread/fwrite/ fopen/fclose open/close How your application reaches H.D.D. !18 Device Driver Device Driver Device

• How do we locate files? • How do we manage hierarchical namespace? • How do we manage file and file system metadata?

• How do we allocate storage space? • How do we make the file system fast? • How do we ensure file integrity?

!30

Questions for file systems

Page 28: Hung-Wei TsengFile system Device independent I/O interface (e.g. ioctl) fread/fwrite/ fopen/fclose open/close How your application reaches H.D.D. !18 Device Driver Device Driver Device

How the original UNIX file system use these blocks

!31

tracksector

cylinder

0 78 15

16 2324 3132 3940 4748 5556 63

Disk blocksFile System Metadata (Superblock)

Information about the “file system” itself.(e.g. free blocks)

File Metadata Information about the “files”. e.g. inodes

Data

Data

Page 29: Hung-Wei TsengFile system Device independent I/O interface (e.g. ioctl) fread/fwrite/ fopen/fclose open/close How your application reaches H.D.D. !18 Device Driver Device Driver Device

• Contains critical file system information • The volume size • The number of nodes • Pointer to the head of the free list

• Located at the very beginning of the file system

!32

Superblock — metadata of the file system

Page 30: Hung-Wei TsengFile system Device independent I/O interface (e.g. ioctl) fread/fwrite/ fopen/fclose open/close How your application reaches H.D.D. !18 Device Driver Device Driver Device

• File types: directory, file • File size • Permission • Attributes

!33

inode — metadata of each file

Page 31: Hung-Wei TsengFile system Device independent I/O interface (e.g. ioctl) fread/fwrite/ fopen/fclose open/close How your application reaches H.D.D. !18 Device Driver Device Driver Device

• File types: directory, file • File size • Permission • Attributes • Types of pointers:

• Direct: Access single data block • Single Indirect: Access n data blocks • Double indirect: Access n2 data blocks • Triple indirect: Access n3 data blocks

• inode has 15 pointers: 12 direct, 1 each single-, double-, and triple-indirect

• If data block size is 512B and n = 256: max file size = (12+256+2562+2563)*512 = 8GB

!34

Unix inode

Page 32: Hung-Wei TsengFile system Device independent I/O interface (e.g. ioctl) fread/fwrite/ fopen/fclose open/close How your application reaches H.D.D. !18 Device Driver Device Driver Device

• Scenario: User wants to access /home/hungwei/CS202/foo.c

• Procedure: File system will… • Open “/” file (This is in known from superblock.) • Locate entry for “home,” open that file • Locate entry for “hungwei”, open that file • … • Locate entry for “foo.c” and open that file

• Let’s use “strace” to see what happens

!36

What must be done to reach your files

Page 33: Hung-Wei TsengFile system Device independent I/O interface (e.g. ioctl) fread/fwrite/ fopen/fclose open/close How your application reaches H.D.D. !18 Device Driver Device Driver Device

• Contiguous: the file resides in continuous addresses

!38

How do we allocate space?

a.txt

• Non-contiguous: the file can be anywhere

a.txt

• Extents: the file resides in several group of smaller continuous address

a.txt

Page 34: Hung-Wei TsengFile system Device independent I/O interface (e.g. ioctl) fread/fwrite/ fopen/fclose open/close How your application reaches H.D.D. !18 Device Driver Device Driver Device

• Need to track location of blocks on per file basis • Contiguous only needs a pair <start, size> • Extents requires a table of pairs • Non-contiguous requires either a linked list of blocks OR a

table of block pointers (i.e. a map)

!40

Space overhead for storage allocation strategies

Page 35: Hung-Wei TsengFile system Device independent I/O interface (e.g. ioctl) fread/fwrite/ fopen/fclose open/close How your application reaches H.D.D. !18 Device Driver Device Driver Device

• Disk accesses are slow! • Memory access: 100ns • Disk access: 5-12ms • Flash SSD: 30-120us

• Can reduce average access time by clustering data together… but still slow!

• Ideas: Reduce the number of disk accesses using: • Read-ahead: Bring in multiple blocks when reading a single

block (locality!)

!41

Now, what about performance?

Page 36: Hung-Wei TsengFile system Device independent I/O interface (e.g. ioctl) fread/fwrite/ fopen/fclose open/close How your application reaches H.D.D. !18 Device Driver Device Driver Device

• Buffer cache is a cache of recently used disk blocks resides in DRAM-based main memory

• Modern OSs aggressively use free DRAM space for buffer caches

• When accessing disk (read/write), we follow these steps: • Check if block is in cache; stop if in cache • If not in cache, access disk and place block in the cache • Replacement Policy: LRU implemented with a linked list • Head of list is next to replace • Tail of list is last to replace

!42

Buffer Cache


Recommended