1

Chapter 7

Indexing

File Structures by Folk, Zoellick, and Ricarrdi

2

Chapter Objectives

Index files. Operations Required to Maintain an Index File. Primary keys. Secondary keys.

3

Index a tool for finding records in a file consists of:

key field field on which the index is searched

reference (address or RRN) field tells where to find the data file record associated with

a particular key.

7.1 What is an Index

4

Examples of an Index book index

usually at the end of the book arranged alphabetically by topic

The index in a library (an on-line catalog) allows you to locate items by an author, by a title, or by a call number.

photo thumbnails usually represents a link to the actual photo

much smaller file, can be loaded quickly actual photo takes much longer to load

if index was actual photos, would take long to load

5

Book Index

6

Example: Index in Databases

University uses an index file to keep track of its courses.

The data file consists of the following fields in each record: Department Title Professor Student List Room & Time

7

Example: Primary key Department

not specific enough Course Number

not unique Professor

not unique Room & Time

possible classes aren’t identified this way

Department + Course Number -> Obvious?

7.2 A Simple Index

8

Index file It is used to provide rapid access to

individual records in the data file via the keys

Example index file consists of the following fields: key (e.g. CIS402) reference (address) =address of the

corresponding record in the data file

9

10

Primary Index

k1 k2 k4 k5 k7 k9

k1 k2 k4 k5 k7 k9

AAA ZZZ CCC XXX EEE FFF

Index File

Data File

7.1 What Is an Index?

7.2 A Simple Index for E-S Files

Class TextIndex{ public: TextIndex(int maxKeys = 100, int unique = 1);

int Insert(const char*ckey, int recAddr); //add to index int Remove(const char* key); //remove key from index int Search(const char* key) const;

//search for key, return recAddr void Print (ostream &) const; protected: int MaxKeys; // maximum num of entries int NumKeys;// actual num of entries char **Keys; // array of key values int* RecAddrs; // array of record references int Find (const chat* key) const; int Init (int maxKeys, int unique); int Unique;// if true --> each key must be unique}

Index Class Interface

13

Operations on an Indexed File

Create (when data file is created) Load into memory (whole file, if possible and

prudent) Write updated file to permanent storage Record(s) added to data file Record(s) deleted from data file Update record(s) in data file

Searches

14

15

Creating Files of Data Create files

index file data record file

Load Index via

buffer I/O an array.

Writing Back Index File Can be part of the close operation for the index file

close function in index object can write the buffer/array to the disk before closing file

16

Record addition Adding a new data record to the data file

requires adding a new record to the index file If the index file is sorted:

adding a new record may require rearranging the records in this file.depends upon index file representation in memory if sort necessary, easily done if the indices are in main

memory

17

Record deletion Deletion of a data record requires deletion of the

corresponding index record. Can space in data file be reclaimed?

Difficult, as with index file organization all data records are pinned a pinned data record is one that has a reference to its address in

an index file

Other consequences Resorting difficulty

Solution: Sort the file via the indices

18

Record Updating Two categories of updates:

modification of key value re-ordering of the index file might be required

two possible situations1. modifying key reorders file

2. see below

modification of non-key value might still require reordering of records in the data

file. (WHY?)

size of data record might increase, requiring moving it to space that can hold it

must reset index for that record

19

Indexes too large for Memory

kept on the secondary storage disadvantages

time consumptionsearching the index file

requires disk accesses instead of just memory accesses

rearranging indexesrequires disk accesses

7.5 Indexes That Are Too Large to Hold in Memory

20

Solutions to Index Files in 2ndary storage

If the index file is too large to be kept in main memory than the following alternative organizations should be considered:

a hashed organization (if access speed is very important)

a tree structured organization, or a multilevel index such as a B-tree

21

Pros of a simple index file allows for use of binary search sorting and maintaining an index is much

easier than for a data file true if index entries are much smaller than data

records,

if data records are pinned, can rearrange keys without moving data records

apply them to multiple simple indexes...

22

Indexing with Multiple Key Access

unique primary key often used as a search keyword. Example primary key

CS215

What if you’d like to include the prof in the search? Two keys: Course & Prof Could also be:

Course & Time Location & Time (?)

7.6 Indexing to Provide Access by Multiple Keys

23

Secondary key A secondary key is a key for

which multiple records may exist in the data file.

Example: Sorting an Excel sheet using two

fields (e.g. name & section) A professor teaches more than

one class

24

Secondary Index File create for the possible secondary indexes.

secondary keys can be sharedprimary keys were unique

Example: Professor El-Ramly secondary keys:

Primary keys containing this prof: CS352 CS215

Can access those courses via the secondary key What if course has multiple sections?

25

Record Addition Adding a record to the data file likely requires

adding a record to the secondary index file. Costs are similar to the cost of adding a record

in the primary index file. records might have to be shifted indexes may have to be rearranged

26

27

Record Deletion must remove all references to that record in the file system. search for primary key in primary index file

remove index search in secondary index file

for the primary key of the record to be deleted remove index from the secondary index file.

what if secondary keys are maintained? secondary key refers to primary key primary key will have been deleted, and will not exist if we consider this possibility, don’t have to delete secondary key pitfalls?

28

Record Updating There are three possible situations:

secondary key altered may have to rearrange the secondary key index so it stays in sorted

order primary key altered

big impact on the primary key index in the secondary key index only need to update the affected primary

key field confined to non-key fields

all updates that do not affect either the primary or secondary key fields do not affect the secondary key index, even if the update is substantial. recall, can affect primary index, since that refers to location in data file

29

30

Retrieving Data with Multiple Secondary Keys

Example: All courses taught by Spiegel or Gordon Requires two searches searches produce a list of courses by providing

primary keys.Spiegel: CIS136, CIS235, CIS402Gordon: CIS425, CIS520, CIS243

31

Boolean AND in searches Example:

Search for courses: taught by Spiegel located in Lytle Hall

Courses found are in intersection ofcourses taught by Spiegelcourses offered in Lytle Hall

32

Boolean OR searches Example:

Search for courses: taught by Spiegel located in Lytle Hall

Courses found are in union ofcourses taught by Spiegelcourses offered in Lytle Hall

33

Cons of the Current Secondary Index Structure

index file has to be rearranged every time a new record is added to the file.

for duplicate secondary keys, secondary key field is repeated for each entry.

Secondary Key Primary Key

El-Ramly CS215

El-Ramly CS352

Khattab CS214

Khattab CS316

7.8 Improving the Secondary Index Structure

34

Cons of the Current Secondary Index Structure

Solution A: by an array of references

Solution B: by linking the list of references

7.8 Improving the Secondary Index Structure

35

Improvements to the secondary index key

structure Solution 1

Allow for multiple primary keys to be associated with a single secondary key by allocating a primary key list (STL vector is best; why?) for each secondary key entry. Solves the problem of sorting each time when an new entry is

added. According to text: Suffers from internal fragmentation due to

fixed nature of list, and the number of allocated entries in the array may prove too small. STL (or Java) vector fixes this: How?

A. Array of References

BEETHOVEN ANG3795 DG139201 DG18807 RCA2626

COREA WAR23699

DVORAK COL31809

PROKOFIEV LON2312

RIMSKY-KORSAKOV MER75016

SPRINGSTEEN COL38358

SWEET HONEY IN THE R FF245

Secondary key Set of primary key references

Revised composer index

* no need to rearrange

* limited reference array

* internal fragmentation

37

Allow for multiple primary keys to be associated with a single secondary key by allocating a primary key array (STL vector is best; why?) for each secondary key entry.

Solves the problem of sorting each time when an new entry is added.

According to text: Suffers from internal fragmentation due to fixed nature of list, and the number of allocated entries in the array may prove too small. STL (or Java) vector fixes this: How?

A. Array of References

38

Solution B Create an inverted list of indexes. Have each

secondary key point to a list of primary key references associated with it.This method eliminates most of the problems

associated with maintaining a secondary index file.

Which solution is better?

B. Inverted List

Inverted Lists

Guidelines for better solution no reorganization when adding no limitation for duplicate key no internal fragmentation

Solution B: by Linking the list of references

A list of primary key references

secondary key field, relative record number of the first

corresponding primary key reference

PROKOFIEV ANG36193

LON2312

Linking List of References (1)

BEETHOVEN

COREA

PROKOFIEV

RIMSKY-KORSAKOV

SPINGSTEEN

SWEET HONEY IN THE R

DVORAK

3

2

7

10

6

4

9

LON2312

RCA2626

ANG23699

COL38358

DG18807

MER75016

COL31809

DG139201

ANG36193

WAR23699

-1

-1

-1

8

-1

1

-1

-1

5

0

0

1

2

3

4

5

6

7

8

9 FF245 -1

Secondary Index file Label ID List file

Improved revision of the composer index

0

1

2

3

4

5

6

10

Linking List of References (2)

The primary key references in a separate, entry-sequenced file

Advantages rearranges only when secondary key changes rearrangement is quick less penalty associated with keeping the secondary index file on

secondary storage (less need for sorting) Label ID List file not need to be sorted reusing the space of deleted record is easy

Linking List of References (3)

Disadvantage same secondary key references may not be

physically grouped lack of localitycould involve a large amount of seekingsolution: reside in memory

same Label ID list can hold the lists of a number of secondary index files

if too large in memory, can load only a part of it

43

Selective Indexes

Selective Index: Index on a subset of records Selective index contains only some part of

entire index provide a selective view useful when contents of a file fall into several

categories e.g. 20 < Age < 30 and $1000 < Salarye.g. Courses offered after 12 noon

7.9 Selective Indexes