Section 6 # 1. 6. Files of (horizontal) Records. The concepts of pages or blocks suffices when doing I/O, but the higher layers of a DBMS operate on records and files of records FILE : A collection of pages, each containing a collection of records, which supports: - PowerPoint PPT Presentation
28
6. Files of (horizontal) Records The concepts of pages or blocks suffices when doing I/O, but the higher layers of a DBMS operate on records and files of records • FILE : A collection of pages, each containing a collection of records, which supports: – insert, delete, and modify (on a record) – read a particular record (specified using Record ID or RID) – scan of all records (possibly with some conditions on the records to be retrieved) Section 6 # 1
Transcript
6. Files of (horizontal) Records
The concepts of pages or blocks suffices when doing I/O, but the higher layers of a DBMS operate on records and files of records
• FILE: A collection of pages, each containing a collection of records, which supports:
– insert, delete, and modify (on a record)
– read a particular record (specified using Record ID or RID)
– scan of all records (possibly with some conditions on the records to be retrieved)
Section 6 # 1
Files Types• The three basic file organizations supported by the File Manager of most DBMSs
are:
• HEAP FILES (files of un-ordered records)
• SORTED or CLUSTERED FILES ( records sorted or clustered on some field(s) )
•
• HASHED FILES (files in which records are positioned based on a hash function on some field(s)
Section 6 # 2
Unordered (Heap) Files
• Simplest file structure contains records in no particular order.
• As file grows and shrinks, disk pages are allocated and de-allocated.
• To support record level operations, DBMS must:– keep track of the pages in a file– keep track of free space on pages– keep track of the records on a page
• There are many alternatives for keeping track of these.
Section 6 # 3
Heap File Implemented as a Linked List
• The header page id and Heap file name must be stored someplace.• Each page contains 2 `pointers’ plus data.
HeaderPage
DataPage
DataPage
DataPage
DataPage
DataPage
DataPage Pages with
Free Space
Full Pages…
…
Section 6 # 4
Heap File Using a Page Directory
• The entry for a page can include the number of free bytes on the page.• The directory is a collection of pages; linked list implementation is just one
alternative.
DataPage 1
DataPage 2
DataPage N
HeaderPage
DIRECTORY(linked list ofHeader blocksContaining pageIDs)
Section 6 # 5
Heap File FactsRecord insert? Method-1: System inserts new records at the End Of File (need Next Open Slot indicator), moves last record into freed slot following a deletion, updates indicator.
- doesn't allow support of the RID or RRN concept. Or a deleted record slot can remain empty (until file reorganized)- If record are only moved into freed slots upon reorganization, then RIDs and RRNs can be supported.
<- page | record |0| record |1 | record |2 | |3 | |4 | |5
|3 | <- next open slot indicator
Section 6 # 6
Heap File FactsRecord insert Method-2: Insert in any open slot. Must maintain a data structure indicating open slots (e.g., bit filter (or bit map) identifies open slots) - as a list or - as a bit_filter
<- page record
record
record 101001 availability bit filter (0 means available)
If we want all records with a given value in particular field, need an "index"Of course index files must provide a fast way to find the particular value entries of
interest (the heap file organization for index files would makes little sense). Index files are usually sorted files. Indexes are examples of ACCESS PATHS.
Section 6 # 7
Sorted File (Clustered File) FactsFile is sorted on one attribute (e.g., using the unpacked record-pointer page-format)Advantages over heap includes: - reading records in that particular order is efficient - finding next record in order is efficient. For efficient "value-based" ordering
(clustering), a level of indirection is useful (unpacked, record-pointer page-format)
When a page fills up and,e.g., a record must be inserted and clustered between (3,1) and (3,5), one solution is to simply place it on an overflow page in arrival order.
Then the overflow page is scanned like an unordered file page, when necessary.
Periodically the primary and overflow pages can be reorganized as an unpacked record-pointer extent to improve sequential access speed (next slide for an example)
Section 6 # 8
Sorted File (Clustered File) FactsReorganizing a Sorted File with several overflow levels. THE BEFORE:
Hash files A hash function is applied to the key of a record to determine which "file
bucket" it goes to ("file buckets" are usually the pages of that file. Assume there are M pages, numbered 0 through M-1. Then the hash function can be any function that converts the key to a number between 0 and M-1 (e.g., for numeric keys, MODM is typically used. For non-numeric keys, first map the non-numeric key value to a number and then apply MODM...) ). Collisions or Overflows can occur (when a new record hashes to a bucket that is already full). The simplest Overflow method is to use separate Overflow pages:
e.g., h(key) mod M
hkey
Primary bucket pages
10
M-1
Overflow pages(asseparate link list)
Overflow pages are allocated if needed (as a separate link list for each bucket. Page#s are needed for pointers)or a shared link list.
2
Overflow pages(asSingle link list)
...
Long overflow chains can develop and degrade performance.– Extendible and Linear Hashing are dynamic techniques to fix this problem.
Section 6 # 10
Other Static Hashing overflow handling methods
e.g., h(key) mod M
hkey
bucket pages
recrec
Overflow can be handled by open addressing also (more commonly used for internal hash tables where a bucket is a allocation of main memory, not a page.
rec
In Open Addressing, upon collision, search forward in the bucket sequence for the next open record slot.
rec
0
1
2
3
4
5
6
rech(rec_key)=1Collision!2? no3? yes
Then to search, apply h.
If not found, search sequentially ahead until found (circle around to search start point)!
Section 6 # 11
Other overflow handling methods
h0(key)
hthen h1
then h2
...
bucket pages
recrec
Overflow can be handled by re-hashing also.
rec
In re-hashing, upon collision, apply next hash function from a sequence of hash functions..
0
1
2
3
4
5
6
rec
Then to search, apply h.
If not found, apply next hash function until found or list exhausted.
These methods can be combined also.
Section 6 # 12
Extendible Hashing
Idea: Use directory of pointers to buckets,• split just the bucket that overflowed• double the directory when needed
Directory is much smaller than file, so doubling it is cheap. Only one page of data entries is split. No overflow page!
Trick lies in how hash function is adjusted!
Section 6 # 13
Exampleblocking factor(bfr)=4(# entries per bucket)
Local depth of a bucket: # of bits used to determine if an entry belongs to bucket
Global depth of directory: Max # of bits needed to tell which bucket an entry belongs to (= max of local depths)
Insert: If bucket is full, split it (allocate 1 new page, re-distribute over those 2 pages).
GLOBAL DEPTH = gd
DATA PAGES
13*00
01
10
11
2
2
2
2
2
LOCAL DEPTH
DIRECTORY
Bucket A
Bucket B
Bucket C
Bucket D
10*
1* 21*
4* 12* 32* 16*
15* 7* 19*
5*
To find the bucket for a new key value, r, take just the last global depth bits of h(r), not all of it! (last 2 bits in this example)
(for simplicity we let h(r)=r here)
E.g., h(5)=5=101binary thus it's in bucket pointed in the directory by 01.
Apply hash function, h, to key value, rFollow pointer of last 2 bits of h(r).
Section 6 # 14
Examplehow did we get there? GLOBAL DEPTH = gd
DATA PAGES
0
1
1
1LOCAL DEPTH
DIRECTORY
Bucket A4*
First insert is 4:
h(4) = 4 = 100binary in bucket pointed to by 0 in the directory.
0
Bucket B
Section 6 # 15
ExampleGLOBAL DEPTH = gd
DATA PAGES
0
1
1
1LOCAL DEPTH
DIRECTORY
Bucket A4*
Insert: 12, 32, 16 and 1
1
Bucket B1*
12* 32* 16*
h(12) = 12 = 1100binary in bucket pointed in the directory by 0.
h(32) = 32 = 10 0000binary in bucket pointed in the directory by 0.
h(16) = 16 = 1 0000binary in bucket pointed in the directory by 0.
h(1) = 1 = 1binary in bucket pointed in the directory by 1.
Section 6 # 16
ExampleGLOBAL DEPTH = gd
DATA PAGES
0
1
1
1LOCAL DEPTH
DIRECTORY
Bucket A4*
Insert: 5, 21 and 13
1
Bucket B1*
12* 32* 16*
13*21*5*
h(5) = 5 = 101binary in bucket pointed in the directory by 1.
h(21) = 21 = 1 0101binary in bucket pointed in the directory by 1.
h(13) = 13 = 1101binary in bucket pointed in the directory by 1.
Section 6 # 17
0
0
1
1
ExampleGLOBAL DEPTH = gd
DATA PAGES
0
1
2
2LOCAL DEPTH
DIRECTORY
Bucket A4*
9th insert: 10
h(10) = 10 = 1010binary in bucket pointed in the directory by 0. Collision!
1
Bucket B1*
12* 32* 16*
13*21*5*
2Bucket C
10*
Split bucket A into A and C.
0
1
Reset one pointer.
Double directory (by copying what is there
and adding a bit on the left).
Redistribute values among A and C (if necessary Not necessary this time since all green bits (2's position bits) are correct:
4 = 10012 = 110032 =10000016 = 10000
10 = 1010
Section 6 # 18
0
0
1
1
ExampleGLOBAL DEPTH = gd
0
1
2
2LOCAL DEPTH
DIRECTORY
Bucket A4*
h(15) = 15 = 1111binary
1
Bucket B1*
12* 32* 16*
13*21*5*
2Bucket C
10*
Split bucket B into B and D.No need to double directory
because the local depth of B is less than the global depth.
0
1
Reset one pointer, and redistribute values among B and D (if necessary, not necessary this time).
DATA PAGES
1Bucket D15*
Reset local depth of B and D
2
2
Inserts: 15, 7 and 19
h(15) = 7 = 111binary
19*7*
h(19) = 15 = 1 0011binary
Section 6 # 19
Insert h(20)=20=10100 Bucket pointed to by 00 is full!
2 2
2
2
LOCAL DEPTH 2
GLOBAL DEPTHBucket A
Bucket B
Bucket C
Bucket D
1* 5* 21*13*
10*
15* 7* 19*
00
01
10
11
4* 12* 32* 16*
Split A.Double directory and reset 1 pointer.
00
01
10
11
0
0
0
0
1
1
1
1
3
Bucket E(`split image'of Bucket A)
3
3
4* 12*
Redistribute contents of A
Section 6 # 20
Points to Note• 20 = binary 10100. Last 2 bits (00) tell us r belongs in
either A or A2, but not which one. Last 3 bits needed to tell which one.– Local depth of a bucket: # of bits used to determine if an entry
belongs to this bucket.
– Global depth of directory: Max # of bits needed to tell which bucket an entry belongs to (= max of local depths)
• When does bucket split cause directory doubling?
– Before insert, local depth of bucket = global depth. Insert causes local depth to become > global depth; directory is doubled by copying it over and `fixing’ pointer to split image page.
– Use of least significant bits enables efficient doubling via copying of directory!)
Section 6 # 21
Comments on Extendible Hashing
• If directory fits in memory, equality search answered with one disk access; else two.
– Directory grows in spurts, and, if the distribution of hash values is skewed, directory can grow large.
– Multiple entries with same hash value cause problems!
• Delete: If removal of data entry makes bucket empty, can be merged with its `split image’.
– As soon as each directory element points to same bucket as its (merged) split image, can halve directory.
Section 6 # 22
Linear Hash File Starts with M buckets (numbered 0, 1, ..., M-1 and initial hash function, h0=modM (or
more general, h0(key)=h(key)modM for any hash ftn h which maps into the integers
Use Chaining to shared overflow-pages to handle overflows.
At the first overflow, split bucket0 into bucket0 and bucketM and rehash bucket0 records using h1=mod2M. Henceforth if h0 yields value0, rehash using h1=mod2M
At the next overflow, split bucket1 into bucket1 and bucketM+1 and rehash bucket1 records using h1=mod2M. Henceforth if h0 yields value1, use h1
...
When all of the original M buckets have been split (M collisions), then rehash all overflow records using h1. Relabel h1 as h0, (discarding the old h0 forever) and start a new "round" by repeating the process above for all future collisions (i.e., now there are buckets 0,...,(2M-1) and h0 = MOD2M).
To search for a record, let n = number of splits so far in the given round, if h0(key) is not greater than n, then use h1, else use h0.
Summary• Hash-based indexes: best for equality searches, cannot support range
searches.
• Static Hashing can lead to performance degradation due to collision handling problems.
• Extendible Hashing avoids performance problems by splitting a full bucket when a new data entry is to be added to it. (Duplicates may require overflow pages.)
– Directory to keep track of buckets, doubles periodically.
– Can get large with skewed data; additional I/O if this does not fit in main memory.
Section 6 # 26
Summary• Linear Hashing avoids directory by splitting buckets round-robin, and using
overflow pages.
– Overflow pages not likely to be long.
– Duplicates handled easily.
– Space utilization could be lower than Extendible Hashing, since splits not concentrated on `dense’ data areas.
• skewed occurs when the hash values of the data entries are not uniform!
