Oracle Berkeley DB

Programmer’s Reference Guide

Release 4.8

.

Legal Notice

This documentation is distributed under an open source license. You may review the terms of this license at:http://www.oracle.com/technology/software/products/berkeley-db/htdocs/oslicense.html

Oracle, Berkeley DB, and Sleepycat are trademarks or registered trademarks of Oracle. All rights to these marks are reserved. Nothird-party use is permitted without the express prior written consent of Oracle.

To obtain a copy of this document's original source code, please submit a request to the Oracle Technology Network forum at:http://forums.oracle.com/forums/forum.jspa?forumID=271

Published 4/12/2010

http://www.oracle.com/technology/software/products/berkeley-db/htdocs/oslicense.htmlhttp://forums.oracle.com/forums/forum.jspa?forumID=271

Table of ContentsPreface .................................................................................................... xx

Conventions Used in this Book ................................................................... xxFor More Information ....................................................................... xx

1. Introduction ............................................................................................ 1An introduction to data management ........................................................... 1Mapping the terrain: theory and practice ...................................................... 1

Data access and data management ....................................................... 2Relational databases ......................................................................... 3Object-oriented databases ................................................................. 4Network databases ........................................................................... 4Clients and servers ........................................................................... 5

What is Berkeley DB? ............................................................................... 6Data Access Services ......................................................................... 7Data management services ................................................................. 7Design .......................................................................................... 8

What Berkeley DB is not ........................................................................... 8Not a relational database ................................................................... 9Not an object-oriented database ........................................................ 10Not a network database ................................................................... 10Not a database server ...................................................................... 10

Do you need Berkeley DB? ....................................................................... 11What other services does Berkeley DB provide? .............................................. 11What does the Berkeley DB distribution include? ............................................ 12Where does Berkeley DB run? ................................................................... 12The Berkeley DB products ........................................................................ 13

Berkeley DB Data Store .................................................................... 13Berkeley DB Concurrent Data Store ...................................................... 13Berkeley DB Transactional Data Store ................................................... 13Berkeley DB High Availability ............................................................. 14

2. Access Method Configuration ...................................................................... 15What are the available access methods? ...................................................... 15

Btree .......................................................................................... 15Hash ........................................................................................... 15Queue ......................................................................................... 15Recno ......................................................................................... 15

Selecting an access method ..................................................................... 15Hash or Btree? ............................................................................... 16Queue or Recno? ............................................................................ 17

Logical record numbers .......................................................................... 17General access method configuration .......................................................... 19

Selecting a page size ....................................................................... 19Selecting a cache size ...................................................................... 20Selecting a byte order ..................................................................... 21Duplicate data items ....................................................................... 22Non-local memory allocation ............................................................. 23

Btree access method specific configuration .................................................. 23

Page iiDB Reference Guide4/12/2010

Btree comparison ........................................................................... 23Btree prefix comparison ................................................................... 25Minimum keys per page .................................................................... 26Retrieving Btree records by logical record number ................................... 26Compression ................................................................................. 28

Custom compression ................................................................. 29Programmer Notes ................................................................... 32

Hash access method specific configuration ................................................... 33Page fill factor .............................................................................. 33Specifying a database hash ................................................................ 33Hash table size .............................................................................. 33

Queue and Recno access method specific configuration .................................... 34Managing record-based databases ....................................................... 34

Record Delimiters .................................................................... 34Record Length ........................................................................ 34Record Padding Byte Value ......................................................... 34

Selecting a Queue extent size ............................................................ 35Flat-text backing files ...................................................................... 35Logically renumbering records ............................................................ 36

3. Access Method Operations .......................................................................... 38Database open ..................................................................................... 39Opening multiple databases in a single file ................................................... 40

Configuring databases sharing a file ..................................................... 41Caching databases sharing a file ......................................................... 41Locking in databases based on sharing a file ........................................... 41

Partitioning databases ............................................................................ 42Specifying partition keys .................................................................. 42Partitioning callback ....................................................................... 44Placing partition files ...................................................................... 45

Retrieving records ................................................................................. 46Storing records ..................................................................................... 46Deleting records ................................................................................... 47Database statistics ................................................................................ 47Database truncation .............................................................................. 47Database upgrade ................................................................................. 47Database verification and salvage .............................................................. 48Flushing the database cache .................................................................... 48Database close ..................................................................................... 49Secondary indexes ................................................................................ 49Foreign key indexes ............................................................................... 53Cursor operations ................................................................................. 57

Retrieving records with a cursor ......................................................... 57Cursor position flags ................................................................. 57Retrieving specific key/data pairs ................................................. 58Retrieving based on record numbers .............................................. 58Special-purpose flags ................................................................ 58

Storing records with a cursor ............................................................. 60Deleting records with a cursor ............................................................ 61Duplicating a cursor ........................................................................ 62

Page iiiDB Reference Guide4/12/2010

Equality Join ................................................................................. 62Example ................................................................................ 63

Data item count ............................................................................. 65Cursor close .................................................................................. 66

4. Access Method Wrapup ............................................................................. 67Data alignment .................................................................................... 67Retrieving and updating records in bulk ....................................................... 67

Bulk retrieval ................................................................................ 67Bulk updates ................................................................................. 69Bulk deletes ................................................................................. 69

Partial record storage and retrieval ............................................................ 70Storing C/C++ structures/objects .............................................................. 72Retrieved key/data permanence for C/C++ ................................................... 74Error support ....................................................................................... 74Cursor stability .................................................................................... 75Database limits .................................................................................... 76Disk space requirements ......................................................................... 76

Btree .......................................................................................... 76Hash ........................................................................................... 77

Specifying a Berkeley DB schema using SQL DDL ............................................. 79Access method tuning ............................................................................ 79Access method FAQ ............................................................................... 81

5. Java API ............................................................................................... 84Java configuration ................................................................................ 84Compatibility ...................................................................................... 85Java programming notes ......................................................................... 85Java FAQ ............................................................................................ 86

6. C# API .................................................................................................. 89Compatibility ...................................................................................... 89

7. Standard Template Library API ..................................................................... 90Dbstl introduction ................................................................................. 90

Standards compatible ...................................................................... 90Performance overhead ..................................................................... 90Portability .................................................................................... 90

Dbstl typical use cases ........................................................................... 91Dbstl examples .................................................................................... 91Berkeley DB configuration ....................................................................... 93

Registering database and environment handles ....................................... 94Truncate requirements .................................................................... 94Auto commit support ....................................................................... 95Database and environment identity checks ............................................ 95Products, constructors and configurations .............................................. 95

Using advanced Berkeley DB features with dbstl ............................................. 96Using bulk retrieval iterators ............................................................. 96Using the DB_RMW flag .................................................................... 97Using secondary index database and secondary containers .......................... 97

Using transactions in dbstl ....................................................................... 97Using dbstl in multithreaded applications ..................................................... 98Working with primitive types ................................................................... 99

Page ivDB Reference Guide4/12/2010

Storing strings .............................................................................. 100Store and Retrieve data or objects of complex types ..................................... 101

Storing varying length objects .......................................................... 101Storing by marshaling objects .................................................... 101Using a DbstlDbt wrapper object ................................................ 102

Storing arbitrary sequences .............................................................. 103The SequenceLenFunct function .................................................. 104The SequenceCopyFunct function ................................................. 104

Notes ........................................................................................ 104Dbstl persistence ................................................................................. 105

Direct database get ....................................................................... 105Change persistence ....................................................................... 106Object life time and persistence ....................................................... 107

Dbstl container specific notes ................................................................. 108db_vector specific notes ................................................................. 108Associative container specific notes ................................................... 109

Using dbstl efficiently ........................................................................... 109Using iterators efficiently ................................................................ 109Using containers efficiently ............................................................. 110

Dbstl memory management .................................................................... 111Freeing memory ........................................................................... 111Type specific notes ....................................................................... 111

DbEnv/Db ............................................................................ 111DbstlDbt .............................................................................. 112

Dbstl miscellaneous notes ...................................................................... 112Special notes about trivial methods .................................................... 112Using correct container and iterator public types ................................... 113

Dbstl known issues ............................................................................... 1138. Berkeley DB Architecture ......................................................................... 115

The big picture ................................................................................... 115Programming model ............................................................................. 118Programmatic APIs ............................................................................... 118

C ............................................................................................. 118C++ .......................................................................................... 118STL ........................................................................................... 119Java ......................................................................................... 119Dbm/Ndbm, Hsearch ...................................................................... 120

Scripting languages .............................................................................. 120Perl .......................................................................................... 120PHP .......................................................................................... 120Tcl ........................................................................................... 120

Supporting utilities .............................................................................. 1209. The Berkeley DB Environment .................................................................... 122

Database environment introduction .......................................................... 122Creating a database environment ............................................................. 123Opening databases within the environment ................................................. 125Error support ..................................................................................... 127DB_CONFIG configuration file .................................................................. 128File naming ....................................................................................... 128

Page vDB Reference Guide4/12/2010

Specifying file naming to Berkeley DB ................................................. 129Filename resolution in Berkeley DB .................................................... 129Examples .................................................................................... 130

Shared memory regions ......................................................................... 131Security ............................................................................................ 132Encryption ........................................................................................ 133Remote filesystems .............................................................................. 134Environment FAQ ................................................................................ 135

10. Berkeley DB Concurrent Data Store Applications ............................................ 136Concurrent Data Store introduction .......................................................... 136Handling failure in Data Store and Concurrent Data Store applications ................ 138Architecting Data Store and Concurrent Data Store applications ........................ 139

11. Berkeley DB Transactional Data Store Applications ......................................... 143Transactional Data Store introduction ........................................................ 143Why transactions? ................................................................................ 143Terminology ...................................................................................... 143Handling failure in Transactional Data Store applications ................................ 145Architecting Transactional Data Store applications ........................................ 146Opening the environment ...................................................................... 150Opening the databases .......................................................................... 153Recoverability and deadlock handling ........................................................ 156Atomicity .......................................................................................... 159Isolation ........................................................................................... 160Degrees of isolation ............................................................................. 163

Snapshot Isolation ......................................................................... 164Transactional cursors ............................................................................ 165Nested transactions ............................................................................. 168Environment infrastructure .................................................................... 169Deadlock detection .............................................................................. 169Checkpoints ....................................................................................... 171Database and log file archival ................................................................. 173Log file removal .................................................................................. 176Recovery procedures ............................................................................ 176Hot failover ....................................................................................... 178Recovery and filesystem operations .......................................................... 179Berkeley DB recoverability ..................................................................... 180Transaction tuning ............................................................................... 182Transaction throughput ......................................................................... 185Transaction FAQ .................................................................................. 187

12. Berkeley DB Replication ......................................................................... 190Replication introduction ........................................................................ 190Replication environment IDs ................................................................... 192Replication environment priorities ............................................................ 192Building replicated applications ............................................................... 193Replication Manager methods .................................................................. 194Base API Methods ................................................................................ 196Building the communications infrastructure ................................................. 197Connecting to a new site ....................................................................... 198Running Replication Manager in multiple processes ........................................ 199

Page viDB Reference Guide4/12/2010

One replication process and multiple subordinate processes ...................... 199Persistence of network address configuration ........................................ 199Programming considerations ............................................................. 200Handling failure ........................................................................... 200Other miscellaneous rules ............................................................... 200

Elections .......................................................................................... 201Synchronizing with a master ................................................................... 203

Delaying client synchronization ......................................................... 203Client-to-client synchronization ........................................................ 203Blocked client operations ................................................................ 204Clients too far out-of-date to synchronize ............................................ 204

Initializing a new site ........................................................................... 205Bulk transfer ...................................................................................... 205Transactional guarantees ....................................................................... 206Master Leases ..................................................................................... 210Clock Skew ........................................................................................ 212Network partitions ............................................................................... 213Replication FAQ .................................................................................. 214Ex_rep: a replication example ................................................................. 216Ex_rep_base: a TCP/IP based communication infrastructure ............................. 217Ex_rep_base: putting it all together .......................................................... 219

13. Application Specific Logging and Recovery ................................................... 220Introduction to application specific logging and recovery ................................ 220Defining application-specific log records .................................................... 221Automatically generated functions ........................................................... 223Application configuration ....................................................................... 226

14. Programmer Notes ................................................................................ 229Signal handling ................................................................................... 229Error returns to applications ................................................................... 229Environment variables .......................................................................... 231Multithreaded applications ..................................................................... 231Berkeley DB handles ............................................................................. 232Name spaces ...................................................................................... 234

C Language Name Space ................................................................. 234Filesystem Name Space .................................................................. 234

Memory-only or Flash configurations ......................................................... 234Disk drive caches ................................................................................. 237Copying or moving databases .................................................................. 237Compatibility with historic UNIX interfaces ................................................. 238Run-time configuration ......................................................................... 238Programmer notes FAQ ......................................................................... 239

15. The Locking Subsystem .......................................................................... 241Introduction to the locking subsystem ........................................................ 241Configuring locking .............................................................................. 242Configuring locking: sizing the system ....................................................... 243Standard lock modes ............................................................................ 245Deadlock detection .............................................................................. 246Deadlock detection using timers .............................................................. 247Deadlock debugging ............................................................................. 248

Page viiDB Reference Guide4/12/2010

Locking granularity .............................................................................. 250Locking without transactions .................................................................. 251Locking with transactions: two-phase locking ............................................... 252Berkeley DB Concurrent Data Store locking conventions .................................. 252Berkeley DB Transactional Data Store locking conventions ............................... 253Locking and non-Berkeley DB applications ................................................... 255

16. The Logging Subsystem .......................................................................... 256Introduction to the logging subsystem ........................................................ 256Configuring logging .............................................................................. 257Log file limits ..................................................................................... 258

17. The Memory Pool Subsystem .................................................................... 259Introduction to the memory pool subsystem ................................................ 259Configuring the memory pool .................................................................. 261

18. The Transaction Subsystem ..................................................................... 262Introduction to the transaction subsystem .................................................. 262Configuring transactions ........................................................................ 263Transaction limits ................................................................................ 264

Transaction IDs ............................................................................ 264Cursors ...................................................................................... 264Multiple Threads of Control ............................................................. 264

19. Sequences .......................................................................................... 265Introduction to sequences ...................................................................... 265

20. Berkeley DB Extensions: Tcl ..................................................................... 266Loading Berkeley DB with Tcl .................................................................. 266

Installing as a Tcl Package ............................................................... 266Loading Berkeley DB with Tcl ........................................................... 266

Using Berkeley DB with Tcl ..................................................................... 267Tcl API programming notes ..................................................................... 267Tcl error handling ................................................................................ 268Tcl FAQ ............................................................................................ 269

21. Berkeley DB Extensions .......................................................................... 270Using Berkeley DB with Apache ................................................................ 270Using Berkeley DB with Perl .................................................................... 271Using Berkeley DB with PHP .................................................................... 271

22. Dumping and Reloading Databases ............................................................. 274The db_dump and db_load utilities ........................................................... 274Dump output formats ........................................................................... 274Loading text into databases .................................................................... 275

23. System Installation Notes ........................................................................ 276File utility /etc/magic information ........................................................... 276Building with multiple versions of Berkeley DB ............................................. 276

24. Debugging Applications .......................................................................... 278Introduction to debugging ...................................................................... 278Compile-time configuration .................................................................... 278Run-time error information .................................................................... 279Reviewing Berkeley DB log files ............................................................... 279

Augmenting the Log for Debugging ..................................................... 283Extracting Committed Transactions and Transaction Status ....................... 283Extracting Transaction Histories ........................................................ 283

Page viiiDB Reference Guide4/12/2010

Extracting File Histories .................................................................. 283Extracting Page Histories ................................................................ 283Other log processing tools ............................................................... 284

25. Building Berkeley DB for the BREW simulator ................................................ 285Building a BREW applet with Berkeley DB library ........................................... 286Building a BREW applet for the physical device ............................................ 286

26. Building Berkeley DB for S60 .................................................................... 287Building Berkeley DB for the S60 Emulator .................................................. 287Building Berkeley DB Library for the Device ................................................. 287Building a S60 application with the Berkeley DB library ................................... 288S60 notes .......................................................................................... 288

27. Building Berkeley DB for UNIX/POSIX .......................................................... 289Building for UNIX/POSIX ........................................................................ 289Configuring Berkeley DB ........................................................................ 290Building a small memory footprint library ................................................... 294Changing compile or load options ............................................................. 295Installing Berkeley DB ........................................................................... 296Dynamic shared libraries ....................................................................... 297Running the test suite under UNIX ............................................................ 298Architecture independent FAQ ................................................................ 299AIX .................................................................................................. 302FreeBSD ............................................................................................ 303HP-UX .............................................................................................. 303IRIX ................................................................................................. 305Linux ............................................................................................... 305Mac OS X .......................................................................................... 306OSF/1 .............................................................................................. 307QNX ................................................................................................ 307SCO ................................................................................................. 308Solaris .............................................................................................. 308SunOS .............................................................................................. 310Ultrix ............................................................................................... 310

28. Building Berkeley DB for Windows ............................................................. 311Building Berkeley DB for 32 bit Windows ..................................................... 311

Visual C++ .NET 2008 ..................................................................... 311Visual C++ .NET 2005 ..................................................................... 311Visual C++ .NET or Visual C++ .NET 2003 .............................................. 312Visual C++ 6.0 .............................................................................. 312Build results ................................................................................ 312

Building Berkeley DB for 64-bit Windows .................................................... 312x64 build with Visual Studio 2005 or newer ........................................... 313x64 build with Visual Studio .NET 2003 or earlier .................................... 313

Building Berkeley DB with Cygwin ............................................................. 313Building the C++ API ............................................................................. 313Building the C++ STL API ........................................................................ 313Building the Java API ............................................................................ 314

Building Java with Visual C++ .NET or above ......................................... 314Building Java with Visual C++ 6.0 ....................................................... 314

Building the C# API .............................................................................. 315

Page ixDB Reference Guide4/12/2010

Building C# with Visual Studio 2005 .................................................... 315Building the Tcl API .............................................................................. 315

Building Tcl with Visual C++ .NET or above ........................................... 316Building Tcl with Visual C++ 6.0 ........................................................ 316

Distributing DLLs ................................................................................. 317Building a small memory footprint library ................................................... 317Running the test suite under Windows ....................................................... 318

Building the software needed by the tests ............................................ 318Visual Studio 2005 or newer ....................................................... 318Visual Studio 2003 .NET or earlier ............................................... 318

Running the test suite under Windows ................................................. 318Windows notes ................................................................................... 319Windows FAQ ..................................................................................... 320

29. Building Berkeley DB for Windows CE .......................................................... 322Building for Windows CE ........................................................................ 322

Building Berkeley DB for Windows CE .................................................. 322eMbedded Visual C++ 4.0 .......................................................... 322Build results ......................................................................... 322

Building Berkeley DB for different target CPU architectures ....................... 322eMbedded Visual C++ 4.0 .......................................................... 323

Windows CE notes ............................................................................... 323Windows CE/Mobile FAQ ........................................................................ 324

30. Building Berkeley DB for VxWorks .............................................................. 326Building for VxWorks 5.4 and 5.5 .............................................................. 326

Building With Tornado 2.0 or Tornado 2.2 ............................................ 326Building for VxWorks 6.x ........................................................................ 327

Building With Wind River Workbench using the Makefile ........................... 327VxWorks notes .................................................................................... 328

Building and Running the Demo Program .............................................. 328Building and Running the Utility Programs ............................................ 328VxWorks 5.4/5.5: shared memory ...................................................... 329VxWorks 5.4/5.5: building a small memory footprint library ...................... 329

VxWorks FAQ ...................................................................................... 32931. Upgrading from previous versions of Berkeley DB ........................................... 332

Library version information .................................................................... 332Upgrading Berkeley DB installations .......................................................... 332

32. Upgrading Berkeley DB 1.85 or 1.86 applications to Berkeley DB 2.0 ..................... 337Release 2.0: introduction ....................................................................... 337Release 2.0: system integration ............................................................... 337Release 2.0: converting applications ......................................................... 338Release 2.0: Upgrade Requirements .......................................................... 339

33. Upgrading Berkeley DB 2.X applications to Berkeley DB 3.0 ................................ 340Release 3.0: introduction ....................................................................... 340Release 3.0: environment open/close/unlink ............................................... 340Release 3.0: function arguments .............................................................. 343Release 3.0: the DB_ENV structure ........................................................... 344Release 3.0: database open/close ............................................................ 345Release 3.0: db_xa_open ....................................................................... 346Release 3.0: the DB structure .................................................................. 346

Page xDB Reference Guide4/12/2010

Release 3.0: the DBINFO structure ............................................................ 347Release 3.0: DB->join ........................................................................... 349Release 3.0: DB->stat ........................................................................... 349Release 3.0: DB->sync and DB->close ......................................................... 349Release 3.0: lock_put ........................................................................... 349Release 3.0: lock_detect ....................................................................... 349Release 3.0: lock_stat ........................................................................... 349Release 3.0: log_register ....................................................................... 350Release 3.0: log_stat ............................................................................ 350Release 3.0: memp_stat ........................................................................ 350Release 3.0: txn_begin .......................................................................... 350Release 3.0: txn_commit ....................................................................... 350Release 3.0: txn_stat ............................................................................ 350Release 3.0: DB_RMW ........................................................................... 350Release 3.0: DB_LOCK_NOTHELD .............................................................. 351Release 3.0: EAGAIN ............................................................................. 351Release 3.0: EACCES ............................................................................. 351Release 3.0: db_jump_set ...................................................................... 351Release 3.0: db_value_set ...................................................................... 352Release 3.0: the DbEnv class for C++ and Java ............................................. 353Release 3.0: the Db class for C++ and Java .................................................. 354Release 3.0: additional C++ changes .......................................................... 355Release 3.0: additional Java changes ........................................................ 355Release 3.0: Upgrade Requirements .......................................................... 355

34. Upgrading Berkeley DB 3.0 applications to Berkeley DB 3.1 ................................ 356Release 3.1: introduction ....................................................................... 356Release 3.1: DB_ENV->open, DB_ENV->remove ............................................. 356Release 3.1: DB_ENV->set_tx_recover ........................................................ 356Release 3.1: DB_ENV->set_feedback, DB->set_feedback .................................. 357Release 3.1: DB_ENV->set_paniccall, DB->set_paniccall .................................. 357Release 3.1: DB->put ............................................................................ 357Release 3.1: identical duplicate data items ................................................. 358Release 3.1: DB->stat ........................................................................... 358Release 3.1: DB_SYSTEM_MEM ................................................................. 359Release 3.1: log_register ....................................................................... 359Release 3.1: memp_register ................................................................... 359Release 3.1: txn_checkpoint ................................................................... 359Release 3.1: environment configuration ..................................................... 359Release 3.1: Tcl API ............................................................................. 360Release 3.1: DB_TMP_DIR ....................................................................... 360Release 3.1: log file pre-allocation ........................................................... 361Release 3.1: Upgrade Requirements .......................................................... 361

35. Upgrading Berkeley DB 3.1 applications to Berkeley DB 3.2 ................................ 362Release 3.2: introduction ....................................................................... 362Release 3.2: DB_ENV->set_flags ............................................................... 362Release 3.2: DB callback functions, app_private field ..................................... 362Release 3.2: Logically renumbering records ................................................. 362Release 3.2: DB_INCOMPLETE .................................................................. 363Release 3.2: DB_ENV->set_tx_recover ........................................................ 363

Page xiDB Reference Guide4/12/2010

Release 3.2: DB_ENV->set_mutexlocks ....................................................... 363Release 3.2: Java and C++ object reuse ..................................................... 364Release 3.2: Java java.io.FileNotFoundException .......................................... 364Release 3.2: db_dump .......................................................................... 364Release 3.2: Upgrade Requirements .......................................................... 364

36. Upgrading Berkeley DB 3.2 applications to Berkeley DB 3.3 ................................ 365Release 3.3: introduction ....................................................................... 365Release 3.3: DB_ENV->set_server ............................................................. 365Release 3.3: DB->get_type ..................................................................... 365Release 3.3: DB->get_byteswapped ........................................................... 365Release 3.3: DB->set_malloc, DB->set_realloc .............................................. 365Release 3.3: DB_LOCK_CONFLICT ............................................................. 366Release 3.3: memp_fget, EIO .................................................................. 367Release 3.3: txn_prepare ....................................................................... 367Release 3.3: --enable-dynamic, --enable-shared ........................................... 367Release 3.3: --disable-bigfile .................................................................. 367Release 3.3: Upgrade Requirements .......................................................... 367

37. Upgrading Berkeley DB 3.3 applications to Berkeley DB 4.0 ................................ 368Release 4.0: Introduction ....................................................................... 368Release 4.0: db_deadlock ...................................................................... 368Release 4.0: lock_XXX ........................................................................... 368Release 4.0: log_XXX ............................................................................ 368Release 4.0: memp_XXX ........................................................................ 369Release 4.0: txn_XXX ............................................................................ 370Release 4.0: db_env_set_XXX .................................................................. 371Release 4.0: DB_ENV->set_server ............................................................. 372Release 4.0: DB_ENV->set_lk_max ............................................................ 372Release 4.0: DB_ENV->lock_id_free ........................................................... 372Release 4.0: Java CLASSPATH environment variable ....................................... 372Release 4.0: C++ ostream objects ............................................................. 373Release 4.0: application-specific recovery .................................................. 373Release 4.0: Upgrade Requirements .......................................................... 3744.0.14 Change Log ............................................................................... 374

Major New Features: ...................................................................... 374General Environment Changes: ......................................................... 374General Access Method Changes: ....................................................... 375Btree Access Method Changes: .......................................................... 375Hash Access Method Changes: ........................................................... 375Queue Access Method Changes: ......................................................... 375Recno Access Method Changes: ......................................................... 375C++ API Changes: .......................................................................... 375Java API Changes: ......................................................................... 375Tcl API Changes: ........................................................................... 376RPC Client/Server Changes: ............................................................. 376XA Resource Manager Changes: ......................................................... 376Locking Subsystem Changes: ............................................................ 376Logging Subsystem Changes: ............................................................ 376Memory Pool Subsystem Changes: ...................................................... 376Transaction Subsystem Changes: ....................................................... 377

Page xiiDB Reference Guide4/12/2010

Utility Changes: ............................................................................ 377Database or Log File On-Disk Format Changes: ....................................... 377Configuration, Documentation, Portability and Build Changes: ................... 377

38. Upgrading Berkeley DB 4.0 applications to Berkeley DB 4.1 ................................ 379Release 4.1: Introduction ....................................................................... 379Release 4.1: DB_EXCL ........................................................................... 379Release 4.1: DB->associate, DB->open, DB->remove, DB->rename ...................... 379Release 4.1: DB_ENV->log_register ........................................................... 381Release 4.1: st_flushcommit ................................................................... 381Release 4.1: DB_CHECKPOINT, DB_CURLSN .................................................. 381Release 4.1: DB_INCOMPLETE .................................................................. 382Release 4.1: DB_ENV->memp_sync ............................................................ 382Release 4.1: DB->stat.hash_nelem ............................................................ 382Release 4.1: Java exceptions .................................................................. 382Release 4.1: C++ exceptions ................................................................... 382Release 4.1: Application-specific logging and recovery ................................... 383Release 4.1: Upgrade Requirements .......................................................... 383Berkeley DB 4.1.24 and 4.1.25 Change Log .................................................. 383

Database or Log File On-Disk Format Changes: ....................................... 383Major New Features: ...................................................................... 384General Environment Changes: ......................................................... 384General Access Method Changes: ....................................................... 385Btree Access Method Changes: .......................................................... 386Hash Access Method Changes: ........................................................... 386Queue Access Method Changes: ......................................................... 387Recno Access Method Changes: ......................................................... 387C++-specific API Changes: ................................................................ 387Java-specific API Changes: ............................................................... 388Tcl-specific API Changes: ................................................................ 388RPC-specific Client/Server Changes: ................................................... 388Replication Changes: ..................................................................... 388XA Resource Manager Changes: ......................................................... 388Locking Subsystem Changes: ............................................................ 389Logging Subsystem Changes: ............................................................ 389Memory Pool Subsystem Changes: ...................................................... 389Transaction Subsystem Changes: ....................................................... 390Utility Changes: ............................................................................ 390Configuration, Documentation, Portability and Build Changes: ................... 390

Berkeley DB 4.1.25 Change Log ................................................................ 39239. Upgrading Berkeley DB 4.1 applications to Berkeley DB 4.2 ................................ 393

Release 4.2: Introduction ....................................................................... 393Release 4.2: Java ................................................................................ 393Release 4.2: Queue access method ........................................................... 394Release 4.2: DB_CHKSUM_SHA1 ............................................................... 395Release 4.2: DB_CLIENT ........................................................................ 395Release 4.2: DB->del ............................................................................ 395Release 4.2: DB->set_cache_priority ......................................................... 395Release 4.2: DB->verify ......................................................................... 396Release 4.2: DB_LOCK_NOTGRANTED ......................................................... 396

Page xiiiDB Reference Guide4/12/2010

Release 4.2: Replication ........................................................................ 396Replication initialization ................................................................. 396Database methods and replication clients ............................................ 397DB_ENV->rep_process_message() ....................................................... 397

Release 4.2: Client replication environments ............................................... 397Release 4.2: Tcl API ............................................................................. 397Release 4.2: Upgrade Requirements .......................................................... 397Berkeley DB 4.2.52 Change Log ................................................................ 397

Database or Log File On-Disk Format Changes: ....................................... 397New Features: ............................................................................. 398Database Environment Changes: ........................................................ 398Concurrent Data Store Changes: ........................................................ 400General Access Method Changes: ....................................................... 400Btree Access Method Changes: .......................................................... 402Hash Access Method Changes: ........................................................... 402Queue Access Method Changes: ......................................................... 403Recno Access Method Changes: ......................................................... 404C++-specific API Changes: ................................................................ 404Java-specific API Changes: ............................................................... 405Tcl-specific API Changes: ................................................................ 406RPC-specific Client/Server Changes: ................................................... 406Replication Changes: ..................................................................... 406XA Resource Manager Changes: ......................................................... 409Locking Subsystem Changes: ............................................................ 409Logging Subsystem Changes: ............................................................ 410Memory Pool Subsystem Changes: ...................................................... 411Transaction Subsystem Changes: ....................................................... 412Utility Changes: ............................................................................ 412Configuration, Documentation, Portability and Build Changes: ................... 413

40. Upgrading Berkeley DB 4.2 applications to Berkeley DB 4.3 ................................ 416Release 4.3: Introduction ....................................................................... 416Release 4.3: Java ................................................................................ 416Release 4.3: DB_ENV->set_errcall, DB->set_errcall ........................................ 417Release 4.3: DBcursor->c_put .................................................................. 417Release 4.3: DB->stat ........................................................................... 417Release 4.3: DB_ENV->set_verbose ........................................................... 417Release 4.3: Logging ............................................................................ 418Release 4.3: DB_FILEOPEN ..................................................................... 418Release 4.3: ENOMEM and DbMemoryException ............................................ 418Release 4.3: Replication ........................................................................ 419Release 4.3: Run-time configuration ......................................................... 419Release 4.3: Upgrade Requirements .......................................................... 419Berkeley DB 4.3.29 Change Log ................................................................ 419

Database or Log File On-Disk Format Changes: ....................................... 419New Features: ............................................................................. 419Database Environment Changes: ........................................................ 420Concurrent Data Store Changes: ........................................................ 421General Access Method Changes: ....................................................... 421Btree Access Method Changes: .......................................................... 422

Page xivDB Reference Guide4/12/2010

Hash Access Method Changes: ........................................................... 423Queue Access Method Changes: ......................................................... 423Recno Access Method Changes .......................................................... 424C++-specific API Changes: ................................................................ 424Java-specific API Changes: ............................................................... 424Tcl-specific API Changes: ................................................................ 425RPC-specific Client/Server Changes: ................................................... 425Replication Changes: ..................................................................... 425XA Resource Manager Changes: ......................................................... 427Locking Subsystem Changes: ............................................................ 427Logging Subsystem Changes: ............................................................ 427Memory Pool Subsystem Changes: ...................................................... 428Transaction Subsystem Changes: ....................................................... 428Utility Changes: ............................................................................ 428Configuration, Documentation, Portability and Build Changes: ................... 429

41. Upgrading Berkeley DB 4.3 applications to Berkeley DB 4.4 ................................ 431Release 4.4: Introduction ....................................................................... 431Release 4.4: DB_AUTO_COMMIT ............................................................... 431Release 4.4: DB_DEGREE_2, DB_DIRTY_READ ............................................... 431Release 4.4: DB_JOINENV ....................................................................... 431Release 4.4: mutexes ........................................................................... 432Release 4.4: DB_MPOOLFILE->set_clear_len ................................................. 432Release 4.4: lock statistics ..................................................................... 433Release 4.4: Upgrade Requirements .......................................................... 433Berkeley DB 4.4.16 Change Log ................................................................ 433

Database or Log File On-Disk Format Changes: ....................................... 433New Features: ............................................................................. 433Database Environment Changes: ........................................................ 434Concurrent Data Store Changes: ........................................................ 435General Access Method Changes: ....................................................... 435Btree Access Method Changes: .......................................................... 436Hash Access Method Changes: ........................................................... 436Queue Access Method Changes: ......................................................... 436Recno Access Method Changes .......................................................... 436C++-specific API Changes: ................................................................ 437Java-specific API Changes: ............................................................... 437Java collections and bind API Changes: ................................................ 437Tcl-specific API Changes: ................................................................ 438RPC-specific Client/Server Changes: ................................................... 438Replication Changes: ..................................................................... 438XA Resource Manager Changes: ......................................................... 439Locking Subsystem Changes: ............................................................ 439Logging Subsystem Changes: ............................................................ 439Memory Pool Subsystem Changes: ...................................................... 440Transaction Subsystem Changes: ....................................................... 440Utility Changes: ............................................................................ 441Configuration, Documentation, Portability and Build Changes: ................... 441

Berkeley DB 4.4.20 Change Log ................................................................ 442Changes since Berkeley DB 4.4.16: ..................................................... 442

Page xvDB Reference Guide4/12/2010

42. Upgrading Berkeley DB 4.4 applications to Berkeley DB 4.5 ................................ 444Release 4.5: Introduction ....................................................................... 444Release 4.5: deprecated interfaces ........................................................... 444Release 4.5: DB->set_isalive ................................................................... 444Release 4.5: DB_ENV->rep_elect .............................................................. 444Release 4.5: Replication method naming .................................................... 445Release 4.5: Replication events ............................................................... 445Release 4.5: Memory Pool API ................................................................. 445Release 4.5: DB_ENV->set_paniccall .......................................................... 445Release 4.5: DB->set_pagesize ................................................................ 446Release 4.5: Collections API ................................................................... 446Release 4.5: --enable-pthread_self ........................................................... 446Release 4.5: Recno backing text source files ................................................ 446Release 4.5: Application-specific logging .................................................... 447Release 4.5: Upgrade Requirements .......................................................... 447Berkeley DB 4.5.20 Change Log ................................................................ 447

Database or Log File On-Disk Format Changes: ....................................... 447New Features: ............................................................................. 447Database Environment Changes: ........................................................ 447Concurrent Data Store Changes: ........................................................ 448General Access Method Changes: ....................................................... 448Btree Access Method Changes: .......................................................... 449Hash Access Method Changes: ........................................................... 449Queue Access Method Changes: ......................................................... 449Recno Access Method Changes: ......................................................... 449C++-specific API Changes: ................................................................ 449Java-specific API Changes: ............................................................... 450Java collections and bind API Changes: ................................................ 450Tcl-specific API Changes: ................................................................ 450RPC-specific Client/Server Changes: ................................................... 450Replication Changes: ..................................................................... 450XA Resource Manager Changes: ......................................................... 451Locking Subsystem Changes: ............................................................ 451Logging Subsystem Changes: ............................................................ 451Memory Pool Subsystem Changes: ...................................................... 451Transaction Subsystem Changes: ....................................................... 451Utility Changes: ............................................................................ 452Configuration, Documentation, Portability and Build Changes: ................... 452

43. Upgrading Berkeley DB 4.5 applications to Berkeley DB 4.6 ................................ 454Release 4.6: Introduction ....................................................................... 454Release 4.6: C API cursor handle method names ........................................... 454Release 4.6: DB_MPOOLFILE->put ............................................................. 454Release 4.6: DB_MPOOLFILE->set ............................................................. 455Release 4.6: Replication Events ............................................................... 455Release 4.6: DB_REP_FULL_ELECTION ........................................................ 455Release 4.6: Verbose Output ................................................................... 456Release 4.6: DB_VERB_REPLICATION .......................................................... 456Release 4.6: Windows 9X ....................................................................... 456Release 4.6: Upgrade Requirements .......................................................... 456

Page xviDB Reference Guide4/12/2010

Berkeley DB 4.6.21 Change Log ................................................................ 4574.6.21 Patches: ............................................................................ 4574.6.19 Patches ............................................................................. 457Database or Log File On-Disk Format Changes: ....................................... 457New Features: ............................................................................. 457Database Environment Changes: ........................................................ 458Concurrent Data Store Changes: ........................................................ 459General Access Method Changes: ....................................................... 459Btree Access Method Changes: .......................................................... 460Hash Access Method Changes: ........................................................... 460Queue Access Method Changes: ......................................................... 460Recno Access Method Changes: ......................................................... 460C++-specific API Changes: ................................................................ 461Java-specific API Changes: ............................................................... 461Java collections and bind API Changes: ................................................ 461Tcl-specific API Changes: ................................................................ 461RPC-specific Client/Server Changes: ................................................... 462Replication Changes: ..................................................................... 462XA Resource Manager Changes: ......................................................... 463Locking Subsystem Changes: ............................................................ 463Logging Subsystem Changes: ............................................................ 463Memory Pool Subsystem Changes: ...................................................... 463Transaction Subsystem Changes: ....................................................... 463Utility Changes: ............................................................................ 464Configuration, Documentation, Portability and Build Changes: ................... 464

44. Upgrading Berkeley DB 4.6 applications to Berkeley DB 4.7 ................................ 465Release 4.7: Introduction ....................................................................... 465Release 4.7: Run-time configuration ......................................................... 465Release 4.7: Replication API ................................................................... 465Release 4.7: Tcl API ............................................................................. 465Release 4.7: DB_ENV->set_intermediate_dir ................................................ 466Release 4.7: Log configuration ................................................................ 466Release 4.7: Upgrade Requirements .......................................................... 466Berkeley DB 4.7.25 Change Log ................................................................ 466

Database or Log File On-Disk Format Changes: ....................................... 466New Features: ............................................................................. 466Database Environment Changes: ........................................................ 467Concurrent Data Store Changes: ........................................................ 467General Access Method Changes: ....................................................... 467Btree Access Method Changes: .......................................................... 468Hash Access Method Changes: ........................................................... 468Queue Access Method Changes: ......................................................... 468Recno Access Method Changes: ......................................................... 468C-specific API Changes: .................................................................. 468Java-specific API Changes: ............................................................... 468Direct Persistence Layer (DPL), Bindings and Collections API: ..................... 469Tcl-specific API Changes: ................................................................ 469RPC-specific Client/Server Changes: ................................................... 470Replication Changes: ..................................................................... 470

Page xviiDB Reference Guide4/12/2010

XA Resource Manager Changes: ......................................................... 471Locking Subsystem Changes: ............................................................ 471Logging Subsystem Changes: ............................................................ 471Memory Pool Subsystem Changes: ...................................................... 472Mutex Subsystem Changes: .............................................................. 472Transaction Subsystem Changes: ....................................................... 472Utility Changes: ............................................................................ 472Configuration, Documentation, Sample Application, Portability and BuildChanges: .................................................................................... 473

45. Upgrading Berkeley DB 4.7 applications to Berkeley DB 4.8 ................................ 474Release 4.8: Introduction ....................................................................... 474Release 4.8: Registering DPL Secondary Keys ............................................... 474Release 4.8: Minor Change in Behavior of DB_MPOOLFILE->get .......................... 474Release 4.8: Dropped Support for fcntl System Calls ...................................... 475Release 4.8: Upgrade Requirements .......................................................... 475Berkeley DB 4.8.30 Change Log ................................................................ 475

Changes between 4.8.26 and 4.8.30: .................................................. 475Known bugs in 4.8 ........................................................................ 476Changes between 4.8.24 and 4.8.26: .................................................. 476Changes between 4.8.21 and 4.8.24: .................................................. 476Changes between 4.7 and 4.8.21: ...................................................... 477Database or Log File On-Disk Format Changes: ....................................... 477New Features: ............................................................................. 477Database Environment Changes: ........................................................ 478Concurrent Data Store Changes: ........................................................ 478General Access Method Changes: ....................................................... 478Btree Access Method Changes: .......................................................... 479Hash Access Method Changes: ........................................................... 479Queue Access Method Changes: ......................................................... 479Recno Access Method Changes: ......................................................... 479C-specific API Changes: .................................................................. 480C++-specific API Changes: ................................................................ 480Java-specific API Changes: ............................................................... 480Direct Persistence Layer (DPL), Bindings and Collections API: ..................... 480Tcl-specific API Changes: ................................................................ 482RPC-specific Client/Server Changes: ................................................... 482Replication Changes: ..................................................................... 482XA Resource Manager Changes: ......................................................... 484Locking Subsystem Changes: ............................................................ 484Logging Subsystem Changes: ............................................................ 484Memory Pool Subsystem Changes: ...................................................... 484Mutex Subsystem Changes: .............................................................. 485Test Suite Changes ........................................................................ 485Transaction Subsystem Changes: ....................................................... 486Utility Changes: ............................................................................ 486Configuration, Documentation, Sample Application, Portability and BuildChanges: .................................................................................... 486

46. Test Suite .......................................................................................... 488Running the test suite ........................................................................... 488

Page xviiiDB Reference Guide4/12/2010

Test suite FAQ .................................................................................... 48847. Distribution ........................................................................................ 489

Porting Berkeley DB to new architectures ................................................... 489Source code layout .............................................................................. 491

48. Additional References ............................................................................ 494Additional references ........................................................................... 494

Technical Papers on Berkeley DB ....................................................... 494Background on Berkeley DB Features .................................................. 494Database Systems Theory ................................................................ 495

Page xixDB Reference Guide4/12/2010

PrefaceWelcome to Berkeley DB (DB). This document provides an introduction and usage notes forskilled programmers who wish to use the Berkeley DB APIs.

Conventions Used in this Book

The following typographical conventions are used within in this manual:

Structure names are represented in monospaced font, as are method names. For example:"DB->open() is a method on a DB handle."

Variable or non-literal text is presented in italics. For example: "Go to your DB_INSTALLdirectory."

Program examples are displayed in a monospaced font on a shaded background. For example:

/* File: gettingstarted_common.h */typedef struct stock_dbs { DB *inventory_dbp; /* Database containing inventory information */ DB *vendor_dbp; /* Database containing vendor information */

char *db_home_dir; /* Directory containing the database files */ char *inventory_db_name; /* Name of the inventory database */ char *vendor_db_name; /* Name of the vendor database */} STOCK_DBS;

Finally, notes of interest are represented using a note block such as this.☞For More Information

Beyond this manual, you may also find the following sources of information useful when buildinga DB application:

• Getting Started with Transaction Processing for C [http://www.oracle.com/technology/documentation/berkeley-db/db/gsg_txn/C/BerkeleyDB-Core-C-Txn.pdf]

• Berkeley DB Getting Started with Replicated Applications for C [http://www.oracle.com/technology/documentation/berkeley-db/db/gsg_db_rep/C/Replication_C_GSG.pdf]

• Berkeley DB C API [http://www.oracle.com/technology/documentation/berkeley-db/db/api_reference/C/BDB-C_APIReference.pdf]

• Berkeley DB C++ API [http://www.oracle.com/technology/documentation/berkeley-db/db/api_reference/CXX/BDB-CXX_APIReference.pdf]

• Berkeley DB STL API [http://www.oracle.com/technology/documentation/berkeley-db/db/api_reference/STL/BDB-STL_APIReference.pdf]

Page xxDB Reference Guide4/12/2010

http://www.oracle.com/technology/documentation/berkeley-db/db/gsg_txn/C/BerkeleyDB-Core-C-Txn.pdfhttp://www.oracle.com/technology/documentation/berkeley-db/db/gsg_txn/C/BerkeleyDB-Core-C-Txn.pdfhttp://www.oracle.com/technology/documentation/berkeley-db/db/gsg_txn/C/BerkeleyDB-Core-C-Txn.pdfhttp://www.oracle.com/technology/documentation/berkeley-db/db/gsg_db_rep/C/Replication_C_GSG.pdfhttp://www.oracle.com/technology/documentation/berkeley-db/db/gsg_db_rep/C/Replication_C_GSG.pdfhttp://www.oracle.com/technology/documentation/berkeley-db/db/gsg_db_rep/C/Replication_C_GSG.pdfhttp://www.oracle.com/technology/documentation/berkeley-db/db/api_reference/C/BDB-C_APIReference.pdfhttp://www.oracle.com/technology/documentation/berkeley-db/db/api_reference/C/BDB-C_APIReference.pdfhttp://www.oracle.com/technology/documentation/berkeley-db/db/api_reference/C/BDB-C_APIReference.pdfhttp://www.oracle.com/technology/documentation/berkeley-db/db/api_reference/CXX/BDB-CXX_APIReference.pdfhttp://www.oracle.com/technology/documentation/berkeley-db/db/api_reference/CXX/BDB-CXX_APIReference.pdfhttp://www.oracle.com/technology/documentation/berkeley-db/db/api_reference/CXX/BDB-CXX_APIReference.pdfhttp://www.oracle.com/technology/documentation/berkeley-db/db/api_reference/STL/BDB-STL_APIReference.pdfhttp://www.oracle.com/technology/documentation/berkeley-db/db/api_reference/STL/BDB-STL_APIReference.pdfhttp://www.oracle.com/technology/documentation/berkeley-db/db/api_reference/STL/BDB-STL_APIReference.pdf

• Berkeley DB TCL API [http://www.oracle.com/technology/documentation/berkeley-db/db/api_reference/TCL/BDB-TCL_APIReference.pdf]

• Berkeley DB Programmer's Reference Guide [http://www.oracle.com/technology/documentation/berkeley-db/db/programmer_reference/BDB_Prog_Reference.pdf]

Page xxiDB Reference Guide4/12/2010

http://www.oracle.com/technology/documentation/berkeley-db/db/api_reference/TCL/BDB-TCL_APIReference.pdfhttp://www.oracle.com/technology/documentation/berkeley-db/db/api_reference/TCL/BDB-TCL_APIReference.pdfhttp://www.oracle.com/technology/documentation/berkeley-db/db/api_reference/TCL/BDB-TCL_APIReference.pdfhttp://www.oracle.com/technology/documentation/berkeley-db/db/programmer_reference/BDB_Prog_Reference.pdfhttp://www.oracle.com/technology/documentation/berkeley-db/db/programmer_reference/BDB_Prog_Reference.pdfhttp://www.oracle.com/technology/documentation/berkeley-db/db/programmer_reference/BDB_Prog_Reference.pdf

Chapter 1.  Introduction

An introduction to data management

Cheap, powerful computing and networking have created countless new applications that couldnot have existed a decade ago. The advent of the World-Wide Web, and its influence in drivingthe Internet into homes and businesses, is one obvious example. Equally important, though, isthe shift from large, general-purpose desktop and server computers toward smaller,special-purpose devices with built-in processing and communications services.

As computer hardware has spread into virtually every corner of our lives, of course, softwarehas followed. Software developers today are building applications not just for conventionaldesktop and server environments, but also for handheld computers, home appliances, networkinghardware, cars and trucks, factory floor automation systems, and more.

While these operating environments are diverse, the problems that software engineers mustsolve in them are often strikingly similar. Most systems must deal with the outside world,whether that means communicating with users or controlling machinery. As a result, most needsome sort of I/O system. Even a simple, single-function system generally needs to handlemultiple tasks, and so needs some kind of operating system to schedule and manage controlthreads. Also, many computer systems must store and retrieve data to track history, recordconfiguration settings, or manage access.

Data management can be very simple. In some cases, just recording configuration in a flat textfile is enough. More often, though, programs need to store and search a large amount of data,or structurally complex data. Database management systems are tools tha