• Table of Content s • Index SQLite By Chris Newman Publisher : Sams Publishing Pub Date : November 09, 2004 ISBN: 0-672-32685-X Pages : 336 SQLite is a small, fast, embeddable database. What makes it popular is the combination of the database engine and interface into a single library as well as the ability to store all the data in a single file. Its functionality lies between MySQL and PostgreSQL, however it is faster than both databases. In SQLite, author Chris Newman provides a thorough, practical guide to using, administering and programming this up-and-coming database. If you want to learn about SQLite or about its use in conjunction with PHP this is the book for you.
Transcript
• Table ofContents
• Index
SQLiteBy Chris Newman
Publisher: Sams PublishingPub Date: November 09, 2004
ISBN: 0-672-32685-XPages: 336
SQLite is a small, fast, embeddable database. What makes it popular is thecombination of the database engine and interface into a single library as well asthe ability to store all the data in a single file. Its functionality lies betweenMySQL and PostgreSQL, however it is faster than both databases.
In SQLite, author Chris Newman provides a thorough, practical guide to using,administering and programming this up-and-coming database. If you want tolearn about SQLite or about its use in conjunction with PHP this is the book foryou.
• Table ofContents
• Index
SQLiteBy Chris Newman
Publisher: Sams PublishingPub Date: November 09, 2004
ISBN: 0-672-32685-XPages: 336
CopyrightForewordAbout the AuthorWe Want to Hear from You!Reader ServicesIntroduction Welcome to SQLite
Why Use SQLite?Who This Book Is ForHow the Book Is OrganizedVersions of Software CoveredAdditional Resources
Part I: General SQLite Use Chapter 1. Getting Started
IntroductionFeatures and LimitationsWhen Not to Choose SQLiteLooking at SQLite DatabasesHelp and Support
Chapter 2. Working with DataSQLite BasicsQuerying and Updating the Database
Chapter 3. SQLite Syntax and UseNaming ConventionsCreating and Dropping TablesAnatomy of a SELECT StatementAttaching to Another DatabaseManipulating DataIndexesViewsTriggersWorking with Dates and TimesSQL92 Features Not Supported
Chapter 4. Query OptimizationKeys and IndexesThe EXPLAIN Statement
Part II: Using SQLite Programming Interfaces Chapter 5. The PHP Interface
Configuring PHP for SQLite SupportUsing the PHP SQLite ExtensionWorking with User-Defined FunctionsUsing the PEAR Database Class
Chapter 6. The C/C++ InterfacePreparing to Use the C/C++ InterfaceUsing the C Language InterfaceAdding New SQL Functions
Chapter 7. The Perl InterfacePreparing to Use the SQLite InterfaceAbout the Perl DBIUsing the SQLite DBDAdding New SQL Functions
Chapter 8. The Tcl InterfacePreparing to Use the Tcl InterfaceUsing the Tcl Interface
Chapter 9. The Python InterfacePreparing to Use the Python InterfaceUsing the Python Interface
Part III: SQLite Administration Chapter 10. General Database Administration
The PRAGMA CommandBacking Up and Restoring DataExploring the SQLite Virtual Database EngineAccess to the Database File
Part IV: Appendixes Appendix A. Downloading and Installing SQLite
Obtaining SQLite Appendix B. Command Reference for the sqlite Tool
Dot Commands Appendix C. SQL Syntax Reference
Naming ConventionsSQL Command SyntaxANSI SQL Commands and Features Not Supported
Appendix D. PHP Interface ReferencePredefined ConstantsRuntime ConfigurationFunction Reference
Appendix E. C Interface ReferenceThe Core APIThe Non-Callback APIThe Extended APIAdding New SQL Functions
Appendix F. Perl Interface ReferenceThe Perl DBI
Appendix G. Tcl Interface ReferenceThe Tcl Library
Appendix H. Python Interface ReferenceOpening and Closing a DatabaseCreating User-Defined FunctionsError Handling
Appendix I. The Future of SQLiteSQLite Version 3.0
All rights reserved. No part of this book shall be reproduced, stored in a retrieval system, or transmitted by anymeans, electronic, mechanical, photocopying, recording, or otherwise, without written permission from the publisher.No patent liability is assumed with respect to the use of the information contained herein. Although every precautionhas been taken in the preparation of this book, the publisher and author assume no responsibility for errors oromissions. Nor is any liability assumed for damages resulting from the use of the information contained herein.
Library of Congress Catalog Card Number: 2004901350
Printed in the United States of America
First Printing: November 2004
07 06 05 04 4 3 2 1
Trademarks All terms mentioned in this book that are known to be trademarks or service marks have been appropriatelycapitalized. Sams Publishing cannot attest to the accuracy of this information. Use of a term in this book should not beregarded as affecting the validity of any trademark or service mark.
Warning and Disclaimer Every effort has been made to make this book as complete and as accurate as possible, but no warranty or fitness isimplied. The information provided is on an "as is" basis. The author and the publisher shall have neither liability norresponsibility to any person or entity with respect to any loss or damages arising from the information contained in thisbook.
Bulk Sales Sams Publishing offers excellent discounts on this book when ordered in quantity for bulk purchases or special sales.For more information, please contact
U.S. Corporate and Government Sales 1-800-382-3419 [email protected]
ForewordWhen I first began composing SQLite in late May of 2000, I never imagined that a few years later it would be widelyused in consumer electronics devices and in countless programs and that people I barely know would be writingbooks about it. Like most other open-source software, SQLite grew out of a personal need and was intendedprimarily for my own use. In January of 2000, I was working with a team from General Dynamics on a softwareproject that was connected to an Informix database. When Informix works, it works well, but we were havingproblems getting it to start reliably when the host computer was rebooted. We substituted PostgreSQL for Informixon our development systems, but even that database was a hassle to administer. While I was struggling to deal withthese issues, the idea arose to write a simple SQL database engine that was serverless, read ordinary disk files, andcould be statically linked into the application. I took no action on the idea then. But five months later, I was without acontract for a few months and so I started writing SQLite with the thought that it would be handy the next time asimilar problem appeared. The basic database engine was running within a day and was stable within a couple ofweeks. Development was interrupted for a month when my wife, Ginger, and I traveled to central Africa to visitfriends. After our return and exactly four years ago from the day that I write this, I posted SQLite version 1.0 on theweb.
Version 1 of SQLite attracted a small number of users. But the library was of limited utility because it employedGDBM as a storage back end. GDBM has no transaction support and it is based on hashing, so indices could not beused to optimize queries that are constrained by inequalities. To address these limitations, I began working in myspare time on a replacement B-Tree back end in the spring of 2001. Several months passed. I remember that I wasfixing some last-minute bugs in the new system one morning when Ginger called from work to tell me that an airplanehad just crashed into the World Trade Center in New York. Version 2.0 was released a couple of weeks later, amida zeitgeist of sadness.
The release of SQLite version 2.0 triggered a surge of interest. Within weeks, Christian Werner published the firstSQLite add-on, an ODBC driver. Dozens of other wrappers would soon follow. Over the next 12 months, supportwas added for INTEGER PRIMARY KEY, for VIEWs, and LEFT OUTER JOINs. Dan Kennedy contributedcode to implement trIGGERs. By the fall of 2002, SQLite was in essentially the same form as you find it today. Allthrough that year and since, more and more people began using SQLite in their programs and products. In recentmonths, the SQLite website has received visitors from around three thousand distinct IP addresses per day. Sourcecode downloads average more than 400 per day with almost twice that many binary downloads. Total website trafficis approaching one gigabyte per day. SQLite bindings now exist for over two dozen languages including Perl, Python,PHP, Tcl/Tk, Ruby, Lisp, and Java. SQLite has been incorporated into many popular open-source projects, such asKexi, monotone, Mozilla, and Popfile to name a few. I have been told, in confidence, of many commercial softwareprojects built around the library. SQLite has also been spotted in consumer electronics devices such the PhilipsHDD060 MP3 player and in the D-Link DSM-320 Wi-Fi Media Player. No doubt countless other uses of SQLitehave escaped my notice.
From its inception, the primary design goal of SQLite has been simplicity. I have tried to keep SQLite simple toadminister, simple to operate, simple to program, and simple to maintain and customize. Many people tell me that theylike SQLite because it is small, fast, and reliable. Reliability is a consequence of simplicitywith less complication, thereis less to go wrong. Small size and fast performance are just happy accidents. Simplicity is the ultimate goal. In thisway, SQLite is different from enterprise-class database engines that give you most everything you could ever want,except for simplicity. SQLite may have fewer features, but it is much simpler to use and operate, which is moreimportant than a rich feature set in many situations. This is not to say that SQLite will not add new features over time.SQLite will continue to advance and growwe are currently looking at adding support for ALTER TABLE and forforeign keys, for examplebut as long as I control its development, SQLite will continue to be as simple as I can makeit.
Except for a few lines here and there, most of the code in SQLite was written by me and Dan Kennedy. But we havenot been working in isolation. Suggestions and criticisms from the SQLite user community have been invaluable inhelping to direct the library's progress. And though no outside code has been copied into SQLite, other software hasbeen essential in the process of building and testing SQLite. Special thanks go to John Ousterhout and his Tclscripting language. We would have never been able to get SQLite working had it not been for the Tcl language, whichallowed us to write over 33,000 lines of regression test scripts. Other infrastructure software has also played a keyrole. There would be no Windows port of SQLite except for the MinGW build environment. SQLite would nevereven have been started but for the innumerable hordes of developers that have given us X11, GNU, and Linux.SQLite is not a work unto itself but rather a very small and insignificant part of a much larger software ecosystem.
I have formally dedicated my contributions to SQLite to the public domain, and I insist that other contributors dolikewise. (I keep a file of signed copyright releases in the fire safe at my office.) This means that there are no legalrestrictions on copying SQLite. My name appears nowhere in the code. You are free to do with SQLite whateveryou want. I hope that you find SQLite to be helpful and that you will use it for good. You are welcome to use SQLitefor commercial purposes and if you do, I wish you a good profit. Regardless of how you use it, I hope the fact thatyou have received SQLite freely will inspire you to give something back in returnnot to me personally but to thecommunity or to the world as a whole. Finally, and most importantly, I encourage you to find forgiveness for yourown soul and to forgive others in return. Be at peace.
D. Richard HippCharlotte, NCAugust 17, 2004
About the AuthorChris Newman is a consultant programmer specializing in the development of custom web-based databaseapplications to a loyal international client base.
A graduate of Keele University, Chris lives in Stoke-on-Trent, England, where he runs Lightwood Consultancy Ltd,the company he founded in 1999 to further his interest in Internet technology. Lightwood operates web hostingservices under the DataSnake brand and is proud to be one of the first hosting companies to offer and supportSQLite as a standard feature on all accounts.
More information on Lightwood Consultancy Ltd can be found at http://www.lightwood.net, and Chris can becontacted at [email protected].
We Want to Hear from You!As the reader of this book, you are our most important critic and commentator. We value your opinion and want toknow what we're doing right, what we could do better, what areas you'd like to see us publish in, and any otherwords of wisdom you're willing to pass our way.
You can email or write me directly to let me know what you did or didn't like about this bookas well as what we cando to make our books stronger.
Please note that I cannot help you with technical problems related to the topic of this book, and that due to the highvolume of mail I receive, I might not be able to reply to every message.
When you write, please be sure to include this book's title and author as well as your name and phone or emailaddress. I will carefully review your comments and share them with the author and editors who worked on the book.
Reader ServicesFor more information about this book or another Sams title, visit our Web site at www.samspublishing.com. Type theISBN (excluding hyphens) or the title of a book in the Search field to find the page you're looking for.
Introduction Welcome to SQLiteSQLite is one of the fastest-growing database engines around, but that's growth in terms of popularity, not anything todo with its size. In fact one of SQLite's greatest strengths is that it is extremely lightweight indeed yet still manages toretain a large number of features.
Why Use SQLite? There are many reasons for choosing SQLite, including
Performance SQLite performs database operations efficiently and is faster than other free databases such asMySQL and PostgreSQL.
Size SQLite has a small memory footprint and only a single library is required to access databases, making itideal for embedded database applications.
Portability SQLite runs on many platforms and its databases can be ported easily with no client/server setupor ongoing administration required.
Stability SQLite is ACID-compliant, meeting all four criteriaAtomicity, Consistency, Isolation, and Durability.
SQL support SQLite implements a large subset of the ANSI-92 SQL standard, including views, subqueries,and triggers.
Interfaces SQLite has language APIs for C/C++, PHP, Perl, Python, Tcl, and many more beyond thosecovered in this book.
Cost SQLite is in the public domain and therefore is free to use for any purpose without cost and can befreely redistributed.
Who This Book Is For This book is aimed at intermediate- to advanced-level programmers looking to include database functionality withintheir applications. You should have at least a basic working knowledge of one of the languages covered by thisbookC/C++, PHP, Perl, Python, or Tcl. The underlying library is written in C/C++; however, it is not necessary tounderstand that language in order to use the full capabilities of SQLite in your applications.
If you are a new programmer you should still be able to pick up SQLite fairly quickly, but this book does not coverprogramming basics in any of the languages for which there is a SQLite interface. The benefits of SQLite are onlyrealized through using a programming API as it does not include the tools required to operate as a standalonedatabase system.
It is not a prerequisite to have used a relational database in the past. If SQLite will be the first SQL-based databaseyou have encountered, the book gives a thorough SQL tutorial covering the syntax as understood by SQLite. SQLveterans will still benefit from reviewing the sections that cover features included and omitted by SQLite.
How the Book Is Organized This book is organized into three parts.
Part IGeneral SQLite Use
Chapter 1: Getting Started Gives some background on SQLite and discusses its strengths and weaknesses.Covers a few basic SQL commands to get things going and looks at the interactive interfaces to SQLite.
Chapter 2: Working with Data A concise SQLite tutorial introduces working with the SQLite back end.Discusses programming and database design issues related to SQLite.
Chapter 3: SQLite Syntax and Use Examines SQL syntax as supported by SQLite and suggestsworkarounds for the unsupported features.
Chapter 4: Query Optimization Discusses performance considerations related to SQL queries and gives sometechniques that can be used to speed up your database application.
Part IIUsing SQLite Programming Interfaces
Chapter 5: The PHP Interface How to use the SQLite library within PHP scripts to create dynamic,database-driven web pages.
Chapter 6: The C/C++ Interface How to write C/C++ programs using the SQLite library.
Chapter 7: The Perl Interface How to write Perl scripts using the Database Interface module and SQLiteDatabase Driver.
Chapter 8: The Tcl Interface How to write Tcl scripts using the supplied SQLite extension.
Chapter 9: The Python Interface How to write Python programs using the PySQLite extension.
Part IIISQLite Administration
Chapter 10: General Database Administration Discusses basic administration of SQLite and examinesSQLite's internal architecture and the Virtual Database Engine (VDBE).
Versions of Software Covered At the time of writing, the most recent stable version of SQLite is 2.8.15; however, SQLite 3 is already available as abeta and includes some changes and enhancements over the version 2 series. This book was written with theestablished, stable, and well-supported version 2 series in mind.
For the other software discussed in this book, the current versions are as follows:
PHP 5.0.1
Perl 5.8.5
Perl::DBI 1.4.3
DBD::SQLite2 0.32
Tcl 8.4.7
Python 2.3.4
Additional Resources The following are the primary web pages for each of the packages used in this book:
SQLite http://www.sqlite.org/
PHP http://www.php.net/
Perl DBI http://dbi.perl.org/
Tcl http://www.tcl.tk/
Python http://www.python.org/
For support with any of these technologies the relevant mailing list is a good place to start. Instructions on joiningthem can be found in the following locations:
Chapter 1. Getting Started Welcome to SQLite. This chapter will give you an overview of what SQLite is and isn't, what it can and can't do, andhow it compares to other databases that you might consider using or might already be familiar with.
We will look at a few basic SQL commands and see how to use the sqlite command-line tool and the SQLiteDatabase Browser GUI to create, examine, and modify databases.
Introduction SQLite is an embeddable SQL-driven database engine that implements both the database engine and its interface asa C/C++ library. Started in 2000 by D. Richard Hipp, it was written from the ground up and contains absolutely nolegacy code, and the SQLite source code has been in the public domain since the first prerelease of version 2.0 in2001.
The primary design goals when SQLite was conceived were that it should be
Simple to administer
Simple to operate
Simple to use in a program
Simple to maintain and customize
The fact that SQLite is small, fast, and reliablearguably its greatest strengthsis, according to Hipp, a happycoincidence. He concentrated on making SQLite simple, and reliability is a byproduct of having fewer things to gowrong. Having simpler code in the database engine makes it much easier to optimize.
Note
The acronym SQL is sometimes pronounced sequel, although in common usage it is most often said as three letters.SQLite, however, is pronounced sequel-lite by its creatorin the same way that Microsoft SQL Server is usuallypronounced sequel-serverand therefore that is how we have assumed it is said in this book. As we will refer to aSQLite database and an SQL statement, it will help if you are used to hearing them this way as you read on.
SQLite is already widely used in many projects, and its popularity looks set to continue growing. At the time ofwriting, version 3 of SQLite is close to completion and a stable release is expected to be available by the end of2004.
Note
Some of the projects that use SQLite are catalogued at http://www.sqlite.org/cvstrac/wiki?p=SqliteUsers, and youare encouraged to add details of your own projects here too.
Because SQLite 3 has a new API and a new database file format, this book deals primarily with version 2 because itis already available, stable, and well supported. The new features of SQLite 3 are covered in Appendix I, "The Futureof SQLite."
Features and Limitations In this section we will look at the key features of SQLite and some of its limitations. The nature of SQLite makes it anideal choice for quite a number of tasks, but it's not suitable for everything.
It is important to decide whether SQLiteor any other database engine for that matteris the right choice for yourapplication before committing to a particular technology.
Speed
SQLite is extremely efficient, benefiting from a highly optimized internal architecture and a small memory footprint.Because SQLite is not a client/server database, the overheads of running a database daemon and socketcommunication are eliminated.
The published speed comparison at http://www.sqlite.org/speed.html compares SQLite to both MySQL andPostgreSQL. It finds that SQLite can perform up to 20 times faster than PostgreSQL and more than twice as fast asMySQL for common operations.
These tests were performed with default installations of each database, and although it is possible to tune the MySQLand PostgreSQL servers for slightly better performance in a given environment, SQLite does not require any suchoptimization.
The tests found that SQLite is significantly slower than the other databases only on the operations to create an indexand to drop a table. However, slowness in these areas will not affect performance on a production database.
Portability
Because SQLite databases are stored as single files on the filesystem, they are very portable indeed. A database canbe copied from one file to another, even across different operating systems. This means that for a cross-platformdistribution you just need to concentrate on making your code portableeven when a populated database is to beshipped with the application.
SQLite has no external dependencies. The SQLite library is self-contained, so the only system requirement to run anapplication with an embedded SQLite database is the SQLite library itself. Because SQLite can be freely distributed,you can always ensure that this is present.
Security
SQLite databases are stored to the filesystem and access control is performed by the underlying operating systembased on that file's permission settings.
Though SQLite can be accessed by processes running as different users if the correct file permissions are set, thedatabase engine does not detect which user is performing a particular operation.
The advantage of this is one of administrative simplicitythere is no need to set up a complex user grants scheme. Anyuser who has access to read the database file is able to access the database tables and records. Likewise, in a sharedenvironment, users are able to create their own SQLite databases to their file space without any involvement from thesystem administrators.
The disadvantage comes when you want to control permissions at a more finely grained level. There is no GRANToperation that would allow access to particular tables to one set of users but not others. If users have read access,they are able to read the entire database, and if they have write access, you have to be sure of their competence andtrustworthiness with the data!
SQL Implementation
SQLite supports a large subset of the ANSI SQL-92 standard. Some features have a limited implementation and afew features are not supported at all.
For example, atomic transactions are available but cannot be nested; simple subqueries are possible but correlatedsubqueries are not; triggers can fire for each row but not for each statement; views are available but are read-only.
The list of current limitations is maintained at http://www.sqlite.org/omitted.html with the items at the top of the listindicating which items are most likely to be added to SQLite first.
In the vast majority of cases, none of the limitations of SQLite will cause problems when developing your application.For those that you need to work around, the benefits of using a fast, portable embedded database will almostcertainly outweigh the cost of the workaround.
Customization
The SQLite library includes a very powerful mechanism for adding user-defined functions to the SQL command set.Custom functions can even be written in many of the supported language APIs, not just C/C++.
Both simple functions and aggregating functions can be added this way and, as the hard work is done in theapplication code, there is no need to rebuild the SQLite library to add new functionality.
Additionally, as the SQLite source code is public domain, you are free to examine and modify it as you see fit. IfSQLite is missing a feature that you absolutely must have, why not add it yourself and give something back to thecommunity?
Supported APIs
SQLite now has extensive support for other programming languages through APIs that use the underlying C/C++interface to communicate with SQLite database files.
C/C++
The core interface is implemented as a single library called libsqlite.so on Linux/Unix systems and sqlite.dll onWindows.
Only the SQLite library is required to allow all users to create their own databases, so very little administration isrequired to add SQL database capability to a shared system.
PHP
Support for SQLite in PHP has been available for a while, but since the release of PHP 5, it has been shipped withthe standard distribution.
Traditionally, PHP and MySQL have gone hand in hand as the interface and back end for dynamic web sites, but it isexpected that many more web hosting providers will offer SQLite as PHP 5 gains popularity. From the host's point ofview, it is much simpler to administer than a client/server database as there are no complex permissions to manageand database files will be already counted in the disk quota.
Note
SQLite has proved itself to work very well with low- to medium-traffic web sites. As a very rough guideline, if youare expecting over 100,000 hits per dayand in practice, only a small number of web sites will have this level oftrafficyou should consider how much database work is done by the web scripts and think about doing some kind ofstress testing before committing to SQLite as your back end.
Perl
Perl allows access to SQLite through the Database Interface (DBI) module. This makes it very easy for existing Perldevelopers to use SQLite within their scripts. The DBI provides an abstraction layer to the Database Driver (DBD),so the same command set is used to access many different types of underlying databases.
The DBD::SQLite driver further allows access to the capabilities of SQLite that the Perl DBI does not allow.Although this does not create a Perl application that can be easily ported to another database back end, it does allowaccess to the powerful user-defined functions feature.
Tcl
The Tcl interface for SQLite is shipped as part of the SQLite distribution as a library that is imported into Tcl toactivate the extensions.
Using Tcl and Tk together with SQLite, you have a platform that is ideal for rapid development of portable graphicaluser applications.
Other Programming Languages
SQLite has APIs for many other programming languages, including Java, .NET, Smalltalk, and Ruby. As morelanguages become supported, they are added to the list at http://www.sqlite.org/cvstrac/wiki?p=SqliteWrappers.
Scalability
The downside of using a single file to store databases is that SQLite is not as scalable as many client/server databasesystems.
The SQLite engine can address database files of up to 2TB in size; however, the restriction on the size of a databaseis more likely to be enforced by your operating system. In many cases, the size limit on a single file is 2GB.
File locking in SQLite is very coarse-grained. When a write operation takes place, the entire file is locked so that noother process can open it for reading or writing. Larger RDBMSs implement locking at the table or row level so thatother processes are able to carry on working unless they are trying to access a specific piece of locked data.
Therefore, if you have a database that is likely to involve massive database files or a high frequency of slow writeoperations, SQLite may not be suitable and you should consider an RDBMS that is designed and tuned formultiple-user access.
When Not to Choose SQLite There is no hard-and-fast rule that determines whether SQLite is the right choice for your application. It is a robust,fast database engine that implements as much SQL as you are likely to need, but in some situations there are probablybetter choices.
In this section we will look at some circumstances that require features that SQLite cannot provide, followed bysome examples of situations where SQLite is a very good choice indeed.
When SQLite Is Probably Wrong
First the bad news: The following situations may be better handled by another database system.
Network or Client/Server Applications
SQLite is not well suited for multiuser access over a network. SQLite can read from and write to its databases on anetwork share, but performance will take a hit because of the high latency levels that are usually found on a networkfilesystem.
File locking can often be patchy across a network, which could lead to two SQLite processes writing to the databaseat the same time, inevitably causing corruption.
A client/server RDBMS avoids these issues because all filesystem access is performed by the server after its requestshave been received using a network protocol.
High-Volume Websites
For the vast majority of websites SQLite is a good choice. However, some sites receive so much traffic that theirdatabase components would be better suited to a client/server RDBMS.
How to quantify high traffic volume is the million-dollar question. The traffic level itself is not as much of an issue asthe number of database reads and, more importantly, writes that are performed when a typical visitor does somesurfing around.
Because web servers can send multiple requests simultaneously, if the database is frequently locked these processeswill often be hanging around waiting for each other to finish writing before they can do their own thing.
The last thing your busy website needs is a database bottleneck, so you should consider a client/server RDBMS withmore finely grained database locking.
High Concurrency
Similar to the reasons that a high-volume website may not be suitable for SQLite, a multiprocess or multithreadedapplication that performs a large number of database accesses may run into file-locking issues that could be avoidedin an RDBMS that implements locking on smaller subsets of the database.
Again, quantifying a high level of concurrency is difficult, but this is something you should consider if your applicationwill attempt to perform many concurrent database operations.
Each lock on the database file may only last for a few milliseconds, but there are still times where this will be too longfor the application to work efficiently.
When SQLite Is Probably Right
Now for the good news: There are many applications for which SQLite is a great choice.
Websites
We have already mentioned that SQLite is usually a great choice for database-driven websites. With the inclusion ofSQLite support in PHP 5, it is likely to challenge MySQL as the de facto database back end for PHP scripts.
Embedded Devices
Being small and requiring virtually no administration makes SQLite ideal for hardware applications that requiredatabase storage without the need for human intervention. In addition to working on Embedded Linux and WindowsCE, SQLite has been also ported to Palm OS.
Ad-hoc File Storage
If a program writes data to a file on disk, it can write it to a SQLite database just as easily. If you need to writeapplication data to disk, why not use an SQL database rather than devising a custom file format? SQLite has a furtheradvantage over, for instance, XML data files in that the whole file does not have to be read into memory beforespecific elements can be readSQLite provides fast random access to the data.
Internal Data Manipulation
When an application has to perform operations on data held in internal data structures, it can be easier, and oftenquicker, to load that data into an in-memory SQLite database. Then it can be manipulated using SQL commands andreturned to the program.
Looking at SQLite Databases As we have seen, SQLite provides APIs to allow a database to be embedded into a wide variety of programminglanguages. However, there are times when you want to take a look inside your database without writing a program todo so. This section introduces two tools that can be used to query and amend SQLite database files.
The sqlite Tool
SQLite comes bundled with the sqlite tool, which provides a command-line interface to work with database files. It isinvoked from the shell taking simply the name of a database as its argument.
$ sqlite dbfileSQLite version 2.8.15
Enter ".help" for instructions
sqlite>
The sqlite> prompt indicates that sqlite is waiting for a new command to be entered. There are two types ofcommandseither an SQL statement terminated with a semicolon or a command relating to the sqlite program itself,beginning with a dot.
The .help command lists all the dot commands available along with a brief description, as shown in the following:
sqlite> .help.databases List names and files of attached databases
.dump ?TABLE? ... Dump the database in a text format
.echo ON|OFF Turn command echo on or off
.exit Exit this program
.explain ON|OFF Turn output mode suitable for EXPLAIN on or off.
.header(s) ON|OFF Turn display of headers on or off
.help Show this message
.indices TABLE Show names of all indices on TABLE
.mode MODE Set mode to one of "line(s)", "column(s)",
"insert", "list", or "html"
.mode insert TABLE Generate SQL insert statements for TABLE
.nullvalue STRING Print STRING instead of nothing for NULL data
.output FILENAME Send output to FILENAME
.output stdout Send output to the screen
.prompt MAIN CONTINUE Replace the standard prompts
.quit Exit this program
.read FILENAME Execute SQL in FILENAME
.schema ?TABLE? Show the CREATE statements
.separator STRING Change separator string for "list" mode
.show Show the current values for various settings
.tables ?PATTERN? List names of tables matching a pattern
.timeout MS Try opening locked tables for MS milliseconds
.width NUM NUM ... Set column widths for "column" mode
Let's issue a few simple SQL commands to create a new table and add some rows of data. Don't worry too muchabout how these commands work for now; we just need to get something into the database to show how sqliteworks.
First of all, the following statement creates a new table called mytable that has two columns, col1 and col2.
Notice how the prompt changes from sqlite> to three dots, indicating that the entered statement is not yet complete.The semicolon character signifies the end of an SQL statement, and after the semicolon is entered, the promptchanges back.
Note
If you get into a mess entering a statement and want to abort without having SQLite process a nonsense commandthat could potentially do harm, press Ctrl+C to send an interrupt signal to sqlite. You will be returned to the shellprompt and can start again.
Although no output is displayed to screen, SQLite has received and processed the CREATE TABLE command. Theresponse from database operations is always silent unless there has been an error.
If you type the same command with a deliberate mistake, the error will be detected when the semicolon terminator isenteredbut not beforeas shown in the following example:
sqlite> CREATTE TABLE mytable ( ...> col1 NUMERIC, ...> col2 TEXT ...> );SQL error: near "CREATTE": syntax error
The .schema command can be used to see the schema of a table. What is displayed to screen is the CREATETABLE command that was used to create the table. Therefore if you formatted or capitalized the statement differentlythan that shown, the following output will be slightly different.
sqlite> .schema mytableCREATE TABLE mytable (
col1 NUMERIC,
col2 TEXT
);
If .schema is used without a table argument, the schema for all tables in the database is displayed.
The following commands insert a few simple records into the database. Again, the only response after entering eachone is a new sqlite> prompt, unless an error occurs.
sqlite> INSERT INTO mytable VALUES (1, 'One');sqlite> INSERT INTO mytable VALUES (2, 'Two');sqlite> INSERT INTO mytable VALUES (3, 'Three');sqlite> INSERT INTO mytable VALUES (99, 'Ninety Nine');
The SELECT command in SQL is used to retrieve rows from the database. Let's bring back the newly inserted rowsto see how this is displayed in sqlite.
sqlite> SELECT col1, col2 FROM mytable;1|One
2|Two
3|Three
99|Ninety Nine
The sqlite tool has various different output formats for the selected data. The default is list mode which, as shown,outputs one row for each database record with the columns separated by a | character.
To change the separator character, use the .separator command. The following example changes the separator to acomma. You could use this output format to create a comma-separated values file to load into another application, forinstance.
sqlite> .separator ,
sqlite> SELECT col1, col2 FROM mytable;1,One
2,Two
3,Three
99,Ninety Nine
Output modes are changed using the .mode command followed by the name of the desired mode. The column modealso outputs one row for each database record, but uses a formatted column output.
sqlite> .mode column
sqlite> SELECT col1, col2 FROM mytable;1 One
2 Two
3 Three
99 Ninety Nin
By default each column width is 10 characters and, as shown, any values longer than the column width are truncatedto fit. The .width command can be used to adjust this. It takes a series of number arguments with the first numberrepresenting the width of the first column and so on.
Column headings in this mode can be turned on or off using the .header command followed by on or off. Thefollowing example turns headings on and sets the columns widths just wide enough to display all the data and columnheadings.
On systems with the readline library installed, you can re-execute a command that has previously been entered usingthe up and down arrow keys to scroll through the command history a line at a time. In the preceding example it waspossible to resubmit the SELECT statement after entering the two formatting commands by pressing the Up arrowkey three times followed by pressing Enter.
Rather than one line per record, the line output mode causes one row to be output for each selected column with thecolumn name prefixed.
sqlite> .mode linesqlite> SELECT col1, col2 FROM mytable;
col1 = 1
col2 = One
col1 = 2
col2 = Two
col1 = 3
col2 = Three
col1 = 99
col2 = Ninety Nine
Setting output mode to html causes a SELECT statement to generate HTML code for a table that contains thefetched data. The .header setting determines whether column headings are included in <TH> tags.
sqlite> .mode htmlsqlite> .header onsqlite> SELECT col1, col2 FROM mytable;<TR><TH>col1</TH><TH>col2</TH></TR>
<TR><TD>1</TD>
<TD>One</TD>
</TR>
<TR><TD>2</TD>
<TD>Two</TD>
</TR>
<TR><TD>3</TD>
<TD>Three</TD>
</TR>
<TR><TD>99</TD>
<TD>Ninety Nine</TD>
</TR>
The final output mode is insert, which causes a series of SQL INSERT statements to be generated. .mode insertoptionally takes a database table argument, which will generate INSERT statements for the given table name ratherthan the one from which it was selected.
sqlite> .mode insert newtablesqlite> SELECT col1, col2 FROM mytable;INSERT INTO newtable VALUES(1,'One');
INSERT INTO newtable VALUES(2,'Two');
INSERT INTO newtable VALUES(3,'Three');
INSERT INTO newtable VALUES(99,'Ninety Nine');
If you want to check what the current display settings are, use the .show command. The following output shows thecurrent settings if you have followed all the examples in this chapter so far.
sqlite> .show echo: off
explain: off
headers: on
mode: insert
nullvalue:
output: stdout
separator: ,
width: 4 12
The sqlite tool can process both SQL statements and dot commands from standard input. Therefore it is possible topipe or redirect a series of commands to the program rather than key them in.
If you have a script containing a series of SQL commands, for example to create and populate a database, thefollowing command can be used:
$ sqlite dbfile < sqlfile
Alternatively, sqlite uses the .read command to interactively read and process a file containing SQL commands.
sqlite> .read sqlfile
With those basics of sqlite under your belt, you should be all set to start working with SQLite. We will use sqlitethroughout this book and some of the more advanced dot commands will be introduced later on.
SQLite Database Browser
If you prefer to manage your databases through a graphical interface, take a look at SQLite Database Browser. Thishandy open-source application can be built on any platform supported by Trolltech's QT library, and precompiledversions are available for Linux, Windows, and OSX. You can download SQLite Database Browser from http://sourceforge.net/projects/sqlitebrowser/.
When SQLite Database Browser starts up, you will be presented with an empty-looking application window. Let'sload in the sample database that we'll use in Chapter 2, "Working with Data." Download demodb.sql from the SamsPublishing website at http://www.samspublishing.com/. Enter this book's ISBN (without the hyphens) in the Searchbox and click Search. When the book's title is displayed, click the title to go to a page where you can download thecode and save it locally. Then select File, Import, Database from SQL File, as shown in Figure 1.1.
Figure 1.1. Using SQLite Database Browser to import an SQL file.
[View full size image]
A file selector will appear. Select demodb.sql from the location you saved it to. SQLite Database Browser will ask ifyou want to create a new database to store the imported data. Answer Yes and another file selector will appearwhere you can choose the location of your database file. No file extension is necessary, so simply call the filedemodb.
The program will then go ahead and load data from the SQL file to create a new database. Five table names will beshown under the Database Structure tab, and by clicking the + symbols to the left of each one you can expand theview to show the columns in each table.
Note
SQLite Database Browser can also load data from a CSV file into a SQLite database. Select File, Import, Tablefrom CSV File.
Figure 1.2 shows the contents of demodb fully expanded. Notice that the Object column indicates the type of eachitem, and the schema is shown alongside each table object. Unfortunately where the CREATE TABLE statement wasformatted across multiple lines, the schema cannot fit all the information into a single row of the display and sometimesthis looks messy. Expanding the table to display its columns gives as much information as you would need anyway.
Figure 1.2. Viewing table columns through SQLite Database Browser.
[View full size image]
The icon bar at the top of the window allows you to create and drop both tables and indexeshover over each icon inturn and the bubble help will tell you what each one does. These options are also available from the Edit menu.SQLite Database Browser also provides a Modify Table option, which can be very useful as SQLite does notimplement the ALTER TABLE database command.
Note
In the current version of SQLite Database Browser, there is an intermittent problem with the import process.Sometimes after you load a database from an SQL or CSV file, the icon bar and menu options to manipulate tablesare disabled. If this is the case, you need to close the program and reopen your database filethere is no need toreimport it.
To add a new table, click the appropriate icon or select Edit, Create Table. Enter a table name and use the Addbutton to add columns to it. The dialog box will be similar to Figure 1.3.
Figure 1.3. Adding a new table using SQLite Database Browser.
The field type selection has only three valuestext, numeric, and blob. As we'll see in Chapter 2, SQLite is typeless, sothere is not the range of data types you may be used to having in other database systems.
The Browse Data tab enables you to look at the actual records held in the database. Use the Table drop-down list toselect a table and its records will be shown in a grid format below.
Figure 1.4 shows the records in the employees table viewed through SQLite Database Browser.
Figure 1.4. Viewing database records using SQLite Database Browser.
[View full size image]
If there are a large number of rows in the table, the first thousand will be displayed with a scrollbar on the right of thewindow. The buttons in the bottom left corner of the window allow you to flip forward and backward a thousand at atime.
SQLite Database Browser also allows you to narrow the data set displayed using a search option. Click themagnifying glass icon to bring up the Find screen. From here you can select a column to search on, pick an operator,and enter the search value.
For example, to find all male employees, select the employees table under the Browse Data tab and click themagnifying glass. In the Find window, select the sex column and the equals (=) operator and enter M as the value.Click the Search button and two rows will be displayed.
Only the record number and the selected column are displayed in this view, and as you searched on sex=M, showingthat the value of the sex column is M for both of the records isn't much use. However, you can still use the BrowseData view with the Find window active, and clicking an entry in the Find window will automatically highlight thecorresponding row in the main application window, as shown in Figure 1.5.
Figure 1.5. Performing a search in SQLite Database browser.
[View full size image]
Through the Execute SQL tab you can enter any SQL command to have it processed by SQLite. The applicationdisplays any error messages, allowing you to modify the statement until it is error-free. Any rows returned by aSELECT statement are displayed in the bottom half of the screen.
Finally, to make sure any changes you have made through SQLite Database Browser are saved to the database file,select File, Save Database. Changes are not saved automatically, although if you exit the program with unsavedchanges you will be asked if you want to save them.
Help and Support Before we get into using SQLite in the following chapters, let's take a look at the support available to you if you runinto problems.
There is some very good online documentation at http://www.sqlite.org/docs.html, which is updated regularly whennew features are added to SQLite.
If you need something more interactive, the SQLite mailing list is a good place to ask questions. Send a blank emailto [email protected] to join the list, or if you'd rather receive a daily digest of messages, send it [email protected]. The mailing list is frequented by the author and many regular users ofSQLite, so you should find someone who is able to give good advice.
If professional support for your database engine is a requirement, this is available from Hipp, Wyrick & Company,Inc. (known as Hwaci for short) for a fee. Support is provided by the author and maintainer of SQLite, D. RichardHipp, so you know you will be talking to an authority when your moment of need arises. More information can befound at http://www.hwaci.com/sw/sqlite/prosupport.html.
Chapter 2. Working with Data We begin this chapter by getting our hands dirty right away with a basic tutorial that will give you an overview of howto work with data in SQLite.
SQLite Basics If you have used SQL with other database systems, the language elements will be familiar to you, but it is still worthfollowing the tutorial to see some of the differences between SQLite's implementation of SQL and the version you areused to.
If you are new to SQL, you will begin to pick up the basics of the language by following the examples, and we will gointo more depth on each topic in the following chapters.
Prerequisites
To follow the examples in this tutorial you will need access to a workstation or space on a server system on whichyou can create a SQLite database.
You will also need the sqlite command-line tool to be installed and in your path. Full installation instructions can befound in Appendix A, "Downloading and Installing SQLite," but for now the quickest way to get started is to use oneof the precompiled sqlite binaries from http://www.hwaci.com/sw/sqlite/download.html.
Remember, SQLite writes its databases to the filesystem and does not require a database server to be running. Thesingle executable sqlite (or sqlite.exe on Windows systems) is all that you need.
Obtaining the Sample Database
All of the code in this book is available for download on the Sams Publishing website at http://www.samspublishing.com/. Enter this book's ISBN (without the hyphens) in the Search box and click Search.When the book's title is displayed, click the title to go to a page where you can download the code to save you fromretyping all the CREATE TABLE commands that follow. However, you don't have to download the sample databaseas we have included the full set of commands required for the tutorial in this book.
Creating and Connecting to a Database
SQLite stores its databases in files, and you should specify the filename or path to the file when the sqlite utility isinvoked. If the filename given does not exist, a new one is created; otherwise, you will connect to the database thatyou specified as the filename.
$ sqlite demodbSQLite Version 2.8.12
Enter ".help" for instructions
sqlite>
Even without executing any SQLite commands, without any tables being created, connecting to a database asindicated in the preceding example will create an empty database file with the name specified on the command line.
$ ls l-rw-r--r-- 1 chris chris 0 Apr 1 12:00 demodb
No file extension is required or assigned by sqlite; the database file is created with the exact name specified. So howdo we know that this is a SQLite database? You could opt for a consistent file extension, for example usingsomename.db (as long as .db is a unique extension for your system) or use a well-organized directory structure toseparate your data from your program files.
When there is something inside your database, you can identify the file as a SQLite database by taking a look at thefirst few bytes using a binary-safe paging program such as less. The first line you will see will look like this:
** This file contains an SQLite 2.1 database **
We'll examine the format of the database file in more detail later in this chapter, but for now there's nothing more youneed to know about creating and connecting to databases in SQLite. The CREATE DATABASE statement that youmay be used to is not used.
Executing SQL from a File
The sample database is defined by a series of CREATE TABLE statements in a file called demodb.sql and there aresome sample data records in sampledata.sql.
To execute the SQL in these files, use the .read command in sqlite.
$ sqlite demodbSQLite version 2.8.12
Enter ".help" for instructions
sqlite> .read demodb.sql
Nothing will be displayed on screen unless there is an error in the SQL filesilence means that the commands wereexecuted successfully.
It is also possible to execute an SQL file by redirecting it into sqlite from the command line, for instance:
$ sqlite demodb < sampledata.sql
As before, there will only be any output to screen if there are errors in the SQL file.
The Sample Database
The sample database used in this tutorial is a very simple time-tracking system that might be used by a company tolog employees' hours against client projects. There are five tables in the system, with the schema illustrated in Figure2.1.
Figure 2.1. Schema of the sample database.
The tables holding employee and client details are very simple for this tutorial. In a real system the number of fields inthese tables would be much higher, but for simplicity we've limited the employees and clients tables to just the basics.
CREATE TABLE employees (
id INTEGER PRIMARY KEY,
first_name CHAR NOT NULL,
last_name CHAR NOT NULL,
sex CHAR NOT NULL,
email CHAR NOT NULL
);
CREATE TABLE clients (
id INTEGER PRIMARY KEY,
company_name CHAR,
contact_name CHAR,
telephone CHAR
);
Each table has an id field specified as INTEGER PRIMARY KEY, which we'll look at in more detail shortly. In anutshell, this field is used to autogenerate a unique id number for each record added so that we don't have to.
Suppose that the company is working on, or has undertaken in the past, more than one job for each client and wantsto keep the time spent on each project separate. To enable them to associate the hours on a timesheet with aparticular project, we will create a projects table that looks like this:
CREATE TABLE projects (
code CHAR PRIMARY KEY,
client_id INTEGER NOT NULL,
title CHAR NOT NULL,
start_date INTEGER,
due_date INTEGER
);
This time, rather than having SQLite assign our primary key, we've specified it as CHAR so that the company wouldassign its own project codes and employees would know the textual codes of the projects they are working on.
There is no direct relationship between employees and projects or clientswe have assumed that it is possible that anymember of the workforce could be working on any of the current projects. Our timesheets table creates a linkbetween projects and employees when they have done some work and records the date and how long they spent onthat project.
CREATE TABLE timesheets (
id INTEGER PRIMARY KEY,
employee_id INTEGER NOT NULL,
project_code CHAR NOT NULL,
date_worked INTEGER NOT NULL,
hours DECIMAL(5,2) NOT NULL
);
The final table, employee_rates, stores data on how much employees get paid. Rather than a single attribute in theemployees table, this system allows us to remember the previous rates that an employee was paid before any raiseswere given. If we wanted to recalculate the cost of a project from some time ago, we would want to use the rates ofpay in effect at the time and not the present ones.
CREATE TABLE employee_rates (
employee_id INTEGER NOT NULL,
rate DECIMAL(5,2),
start_date INTEGER NOT NULL,
end_date INTEGER
);
Typeless Data Types
The way in which SQLite stores data is built upon the realization that strong typing of data, as found in virtually everyRDBMS on the market, is actually not a useful thing. A database is designed to store and retrieve data, plain andsimple, and as such the developer should not have to declare a column as numeric, textual, or binary, or tie it to anyother specific data type. Needing to do so is a legacy weakness of the underlying system that had to be reflected inSQL, not a feature added to the language.
SQLite is "typeless." It makes no difference whether you put a text string into an integer column or try to shove binarydata into a text type. In fact you canand shouldspecify the column types when each table is created. SQLite's SQLimplementation allows you to do this, but it's actually ignored by the engine.
That said, it's still a good idea to include the column data types in your CREATE TABLE statements to help youthink through the database design thoroughly and as a reminder to yourself and a hint to other programmers as towhat your intended use of each column was. Suppose, also, that in the future you want to migrate or mirror your dataonto an RDBMS that does require column typingyou'll be glad your table definitions are well documented.
You can always retrieve the CREATE TABLE statement used to create a table using the .schema command fromwithin sqlite:
sqlite> .schema employeesCREATE TABLE employees (
id INTEGER PRIMARY KEY,
first_name CHAR NOT NULL,
last_name CHAR NOT NULL,
sex CHAR NOT NULL,
email CHAR NOT NULL
);
In the demo database, all the data types were declared using familiar data type labels, but you may have noticed thatwe did not specify any lengths for the CHAR types, which would usually be required.
In fact the syntax of SQLite's CREATE TABLE statement specifies the column type attribute as optional. A validcolumn type is defined as any sequence of zero or more character strings (other than reserved words) optionallyfollowed by one or two signed integers in parentheses.
Listing 2.1 shows a CREATE TABLE statement with some examples of ANSI SQL data types that are also valid inSQLite.
Listing 2.1. Creating a Table with ANSI SQL Data Types
CREATE TABLE dummy (
mychar CHAR(6),
myvarchar VARCHAR(10)
mydecimal DECIMAL(7,2),
myinteger INTEGER,
mytinyint TINYINT,
myfloat FLOAT,
mydate DATE
);
Listing 2.2 creates a table identical to the one in Listing 2.1, with imaginary column types to show that SQLite pays noattention to the actual types given.
Listing 2.2. Creating a Table with Made-Up Data Types
CREATE TABLE dummy (
mychar CHARACTER(6),
myvarchar VARIABLE LENGTH CHARACTER
mydecimal DECIMAL NUMBER(7,2),
myinteger INT,
mytinyint SMALL NUMBER,
myfloat FLOATING POINT NUMBER,
mydate CALENDAR DATE
);
Finally, Listing 2.3 shows that the same table can actually be created, if you are in a real hurry, without specifying anycolumn types whatsoever.
Listing 2.3. Creating a Table Without Specifying Data Types
CREATE TABLE dummy (
mychar,
myvarchar,
mydecimal,
myinteger,
mytinyint,
myfloat,
mydate
);
The INTEGER PRIMARY KEY
You saw in the preceding section that the words that make up the name of a data type in the CREATE TABLEstatement have no bearing on the data that can be stored in the table. There is actually one exceptionif the column isdefined exactly as INTEGER PRIMARY KEY, the value in the column must be a 32-bit signed integer; otherwise, aninsert instruction will fail.
Note
INTEGER PRIMARY KEY must be used exactly as written here. If it's abbreviated, for instance INT PRIMARYKEY, the column will be created typeless.
So why, with the decision that data typing was a bad idea, has SQLite implemented one column type that's different?The reason for an INTEGER PRIMARY KEY is to allow for the equivalent of an AUTOINCREMENT orIDENTITY column.
You can insert any 32-bit signed integer value into an INTEGER PRIMARY KEY column, or insert a NULL to tellSQLite to assign the value itself. Inserting a NULL will cause a value of one greater than the largest value alreadystored in that column to be used.
If the largest key value is already the maximum value that the column type allows2^311, or 2147483647then it's notpossible to use a greater value for a new key. In this situation, SQLite picks a new key randomly.
We have used an INTEGER PRIMARY KEY called id on the employees, clients, and timesheets tables so that asnew records are added, we do not need to assign this value ourselves.
Working with Dates
In our sample tables, we have declared the date fields as INTEGER. Though SQLite has included some date andtime functions since version 2.8.7, the typeless nature of SQLite does not provide a native storage format for dates. Inour demo database tables, using an integer representation of the date serves our purposes well, for instancetimesheets.date_worked is declared as INTEGER.
The SQLite date functions are based on a string date format, and we will examine them in detail in the next chapter.
Of course, SQLite will not enforce that the value stored in our (typeless) date columns must be an integer, but the useof INTEGER in the CREATE TABLE statement shows our intention for this field.
Using an integer date allows us to use one of two common formats. The UNIX timestamp format is the number ofseconds since midnight on January 1, 1970 (known as the epoch), and a signed 32-bit integer is capable of storingdates up to 2037.
Some date calculations are made very easy using the timestamp format; for instance, to add one day to a value youjust need to add 86400the number of seconds in a dayto it. Most programming languages that SQLite can beembedded in work with this date format, and the SQLite date functions are also able to convert to and from atimestamp.
A more readable integer format is to present a date as YYYYMMDD (or YYYYMMDDHHIISS if the time is alsorequired). It's much easier to see that 20040622 is June 22, 2004, than its equivalent timestamp value of1322222400.
Although the YYYYMMDD format does not make date arithmetic possible without splitting up the individualcomponents or converting to some other format, it can still be used for comparisons because an earlier date willalways be represented as a lower integer value than a later date.
For our sample system, we are storing the dates that an employee worked and a due date for each project, so wemight want to find out the hours worked in a particular month or how many projects are overdue, but we don't needto do any calculations and the time element is not required. Therefore, YYYYMMDD is a suitable date format for thissystem.
Inserting the Sample Data
First of all, we'll add some employees and clients to the system.
INSERT INTO employees (id, first_name, last_name, sex, email)
INSERT INTO clients (id, company_name, contact_name, telephone)
VALUES (501, 'Acme Products', 'Mr R. Runner', '555-6800');
INSERT INTO clients (id, company_name, contact_name, telephone)
VALUES (502, 'ABC Enterprises', 'Mr T. Boss', '555-2999');
INSERT INTO clients (id, company_name, contact_name, telephone)
VALUES (503, 'Premier Things Ltd', 'Mr U. First', '555-4001');
Note
The values in the primary key columns arbitrarily begin at 101 for employees and 501 for clients. Using differentsequences will make it easier to show whether an id number refers to an employee or a client when we are selectingdata in this tutorial. It doesn't actually create a problem if the primary key values of different tables overlap.
Figures 2.2 and 2.3 show the contents of the employees and clients tables after this data has been loaded.
Figure 2.2. Contents of the employees table.
Figure 2.3. Contents of the clients table.
The sample data we have loaded specifies each id value explicitly so that, for instance, we can be sure that projectsbelong to the right client. However, if we add a new client to the system using the following statement, it will be givena new id of the next highest available number:
sqlite> INSERT INTO clients (company_name, contact_name, telephone) ...> VALUES ('Integer Primary Key Corp', 'Mr A. Increment', '555-1234');
We don't have to go trawling through the data to find out what value SQLite assigned for the new primary key. Thefunction last_insert_rowid() will return the integer value that was generated.
sqlite> select last_insert_rowid();504
Next we'll insert details of the clients' projects. The due_date field is optional, and in the case of the second item in thefollowing code being an ongoing project, we give it a NULL value.
INSERT INTO projects (code, client_id, title, start_date, due_date)
Figure 2.4 shows the contents of the projects table after these INSERT statements have been executed. Notice thatthe values in the client_id column correspond to records in the clients table, shown in Figure 2.3.
Figure 2.4. Contents of the projects table.
The following statements insert a rate of pay for each employee. The start_date is the date they joined the companyand in most cases that rate is still in effect. We indicate that a rate is current by the end_date field being NULL.
Debbie Churchill, employee id 104, was given a raise, so there are two entries. Her hours worked before January 1,2004, were paid at $20; from this date onwards she will be paid $25.
INSERT INTO employee_rates (employee_id, rate, start_date, end_date)
VALUES (101, 30.00, 20030401, NULL);
INSERT INTO employee_rates (employee_id, rate, start_date, end_date)
VALUES (102, 15.00, 20020601, NULL);
INSERT INTO employee_rates (employee_id, rate, start_date, end_date)
VALUES (103, 25.00, 20011001, NULL);
INSERT INTO employee_rates (employee_id, rate, start_date, end_date)
VALUES (104, 20.00, 20010401, 20031231);
INSERT INTO employee_rates (employee_id, rate, start_date, end_date)
VALUES (104, 25.00, 20040101, NULL);
Figure 2.5 shows the records inserted into the employee_rates table by the preceding statements. The employee_idcolumn references records from the employees tablethe values in this column correspond to the id column in Figure2.2.
Figure 2.5. Contents of the employee_rates table.
Finally we have some timesheet information. This is only a small sample of information that might be in such a systemwhere new data is added every day, but should be sufficient for demonstration purposes.
INSERT INTO timesheets (employee_id, project_code, date_worked, hours)
VALUES (101, 'ACME01', 20031229, 4);
INSERT INTO timesheets (employee_id, project_code, date_worked, hours)
VALUES (101, 'ABCCONS', 20031229, 2);
INSERT INTO timesheets (employee_id, project_code, date_worked, hours)
VALUES (101, 'PREM3K', 20040102, 6);
INSERT INTO timesheets (employee_id, project_code, date_worked, hours)
VALUES (102, 'ACME01', 20031229, 3);
INSERT INTO timesheets (employee_id, project_code, date_worked, hours)
VALUES (102, 'PREM4K', 20040102, 5);
INSERT INTO timesheets (employee_id, project_code, date_worked, hours)
VALUES (103, 'ABCCONS', 20031229, 2);
INSERT INTO timesheets (employee_id, project_code, date_worked, hours)
VALUES (103, 'PREM4K', 20040102, 3);
INSERT INTO timesheets (employee_id, project_code, date_worked, hours)
VALUES (103, 'ACME01', 20031229, 8);
INSERT INTO timesheets (employee_id, project_code, date_worked, hours)
VALUES (104, 'ACME01', 20040102, 8);
INSERT INTO timesheets (employee_id, project_code, date_worked, hours)
VALUES (104, 'ABCCONS', 20040102, 4);
Figure 2.6 shows the contents of the timesheets table after we have inserted this sample data. Notice that this tablereferences two others, employees and clientsreferenced by employee_id and project_code respectively.
Figure 2.6. The timesheets table after it has had the sample data inserted.
BLOBs and Binary Data
When you define a column as BLOB the column is still typeless, though as you saw earlier this does tell SQLite touse a text-based comparison when sorting on the field. SQLite is fully capable of storing binary data in any column,provided that there are no NUL (0x00, ASCII character zero) characters in the data.
As SQLite will store strings in its columns much more often than binary data, a NUL is the terminator for avariable-length character string and must be encoded somehow if it appears inside a piece of binary data.
Encoding can be done at the application level, for instance by encoding to base-64 or with URL-style encodingwhere NUL becomes its hex value represented as %00, and the percent character itself is encoded to %25.
SQLite also includes a pair of functions to encode and decode binary data, sqlite_encode_binary() andsqlite_decode_binary(). How binary data is encoded for any given column is an implementation choice left to thedeveloper, so you can use supplied functions, adapt them from the sources in src/encode.c, or create your own.
Querying and Updating the Database So now that we have some records in our database, let's look at ways to fetch, modify, and delete records usingSQLite's implementation of SQL.
The SELECT Statement
Use SELECT to fetch rows from a table. Either use a comma-separated list of column names or use an asterisk tofetch all columns in the table.
sqlite> SELECT first_name, email FROM employees;Alex|[email protected]
sqlite> SELECT * FROM clients;501|Acme Products|Mr R. Runner|555-6800
502|ABC Enterprises|Mr T. Boss|555-2999
503|Premier Things Ltd|Mr U. First|555-4001
You'll notice that the format of the output isn't as readable as it could be. The sqlite program has a number offormatting options of which this mode, a pipe-separated list, is the default. The various output modes were discussedin Chapter 1, "Getting Started."
Although character-separated output can be a useful format for parsing by a program, a tabulated output is clearer inprint. Therefore, for the rest of this tutorial we will use column mode with headings turned on.
The values of company_name and contact_name in the preceding code are truncated. Their full values are stored inthe database, but the column display format causes the output to be arbitrarily limited. You can specify the widthsettings in sqlite with the .width command:
504 Integer Primary Key Ltd Mr A. Increment 555-1234
For the rest of this tutorial, display widths have been adjusted to suit the output and the actual .width commandsissued are not shown in the code.
The WHERE Clause
The WHERE statement specifies a conditional clause that can be used to restrict the dataset that is returned. To findall projects for Premier Things, we restrict the dataset on client_id.
sqlite> SELECT code, title ...> FROM projects ...> WHERE client_id = 503;code title
---------- ----------
PREM3K Thing 3000
PREM4K Thing 4000
PREM5K Thing 5000
The AND and OR operators extend a WHERE condition logically. For instance, to find all projects for this client thatwere due on or before August 1, 2003, we could do the following:
sqlite> SELECT code, title, due_date ...> FROM projects ...> WHERE client_id = 503 ...> AND due_date <= 20040801;code title due_date
---------- ---------- ----------
PREM3K Thing 3000 20040430
PREM4K Thing 4000 20040731
You can use the BETWEEN operator to test if a value is in a given range. The following example finds all projects forPremier Things that were due during the month of July 2003.
sqlite> SELECT code, title, due_date ...> FROM projects ...> WHERE client_id = 503 ...> AND due_date BETWEEN 20040701 AND 20040731;code title due_date
---------- ---------- ----------
PREM4K Thing 4000 20040731
You can use the IN operator to test if one value is in another given set of values. To select rows only where the valueis not in the set, use NOT IN. The following example finds all projects for two specified clients:
sqlite> SELECT code, title ...> FROM projects ...> WHERE client_id in (501, 502);code title
---------- -------------------
ACME01 Ordering system
ABCCONS Ongoing consultancy
We will look at the other relational operators that can be used in SQLite in Chapter 3, "SQLite Syntax and Use." Afull list can be found in Table 3.1.
String Comparisons
The examples you have seen so far in this chapter all use integer values in the WHERE clause, and although SQLite istypeless, string values must still be enclosed in single quotes. The following are examples of some string comparisons:
sqlite> SELECT contact_name FROM clients ...> WHERE company_name = 'Acme Products';contact_name
-------------
Mr R. Runner
Note that the equals operator for string comparisons is case sensitive. To find a case-insensitive match, use the LIKEoperator. To test for inequality in strings, use != or NOT LIKE respectively.
sqlite> SELECT company_name FROM clients ...> WHERE company_name LIKE 'acME proDUCTs';company_name
--------------
Acme Products
The LIKE operator can also be used to specify a substring search. The underscore character is used to represent anycharacter, and a percent sign is a variable-length wildcard.
sqlite> SELECT company_name FROM clients ...> WHERE company_name LIKE 'A%';company_name
---------------
Acme Products
ABC Enterprises
Table 2.1 shows some examples of wildcard matching using LIKE.
Table 2.1. Examples of Wildcard Matching with LIKE
Expression Will Match
LIKE 'a%' apple
An apple a day
LIKE 'b_' be
BC
LIKE 'a%a' agenda
ABBA
LIKE 'b_b' Bob
B2B
NULL Values
It is important to remember that NULL is a special value, actually meaning that an item has no value. It is not a zerovalue or an empty string, and comparing either of these values to NULL will always return false. Because NULL hasno value, the equals operator cannot be used and you must test for NULL values using IS NULL.
To check whether a column value is NULL, use the IS NULL operator. The opposite, IS NOT NULL, will matchany value including zero and an empty string. The following query finds all projects that are ongoing, that is with nodue date:
sqlite> SELECT code, title FROM projects ...> WHERE due_date IS NULL;code title
---------- -------------------
ABCCONS Ongoing consultancy
SQLite includes the function ifnull(), which can be used to specify an alternate to be used when a column value isNULL. For instance, we can visually indicate the ongoing projects in the database using this query:
Likewise in a WHERE clause, we can select all the rates of pay that were current as of December 1, 2003, as in thefollowing example:
sqlite> SELECT employee_id, rate ...> FROM employee_rates ...> WHERE start_date <= 20031201 ...> AND ifnull(end_date, 20031201) >= 20031201;employee_id rate
----------- -----
101 30.00
102 15.00
103 25.00
104 20.00
The WHERE clause compares each start_date to see if the rate was valid on or before December 1, and checks thatend_date falls on or after this date. For a row where end_date is NULL, the comparison is between 20031201 and20031201, which will always be true.
Note
SQLite has tried to implement the way it handles NULL in a standards-compliant way; however, in somecircumstances the standards do not clearly dictate how a NULL should he handled. For details of how SQLitehandles NULL compared to other SQL engines, see http://www.hwaci.com/sw/sqlite/nulls.html.
Arithmetic Functions
Simple arithmetic functions in SQLite can be used in the column list of a SELECT statement or in the WHEREclause. In fact, to demonstrate the basic functions you don't even need to select from a table.
sqlite> SELECT 2 + 2;2 + 2
----------
4
sqlite> SELECT 10 - 7;10 - 7
----------
3
sqlite> SELECT 6 * 3;6 * 3
----------
18
sqlite> SELECT 100 / 30;100 / 30
----------
3
Note in the division example that a rounded integer is returned. To specify that you want a floating-point result, one ofthe arguments must be a float itself.
sqlite> SELECT 100.0 / 30;100.0 / 30
----------------
3.33333333333333
The modulo operator is also supported, using the percent symbol, as in many other languages:
sqlite> SELECT 100 % 30;100 % 30
----------
10
You can apply such operators to a selected column in the SELECT statement. For instance, to find all the currentrates of pay plus 15%, we could do this:
sqlite> SELECT rate * 1.15 FROM employee_rates ...> WHERE end_date IS NULL;rate * 1.15
-----------
34.5
17.25
28.75
28.75
Table 2.2 lists the arithmetic operators.
Table 2.2. Arithmetic Operators in Order of Precedence
Operator Meaning
/ Division
* Multiplication
% Modulo
+ Addition
- Subtraction
String Operators and Functions
SQLite's concatenation operator is ||, which can be used to join two or more strings together.
A number of built-in functions are also available for string manipulation. length() takes a single string parameter andreturns its length in characters:
sqlite> SELECT company_name, length(company_name) ...> FROM clients;company_name length(company_name)
------------------------ --------------------
Acme Products 13
ABC Enterprises 15
Premier Things Ltd 18
Integer Primary Key Corp 24
To extract a section of a string, use substr(). Three parameters are requiredthe string itself, the start position (using 1for the leftmost character), and a length:
To capitalize a string or convert it to lowercase, use upper() and lower() respectively:
sqlite> SELECT company_name, upper(company_name) ...> FROM clients;company_name upper(company_name)
------------------------ -------------------
Acme Products ACME PRODUCTS
ABC Enterprises ABC ENTERPRISES
Premier Things Ltd PREMIER THINGS LTD
Integer Primary Key Corp INTEGER PRIMARY KEY
One of SQLite's powerful features is the ability to add user-defined functions to the SQL command set. We will seehow this is done using the various language APIs in Part II, "Using SQLite Programming Interfaces," of this book.
Joins
The key concept of a relational database structure is that of separating your data into tables that model, generallyspeaking, separate physical or logical entitiesthe process of normalizationand rejoining them in a proper manner whenexecuting a query.
In our sample database, we have tables containing a number of relationships. For instance, each employee hasworked a number of hours, usually over the course of a number of days. The number of hours worked on each day isstored as a separate entry in the timesheets table.
The employee_id field in timesheets (often written as timesheets.employee_id) relates to the id field in employees(employees.id). To correlate each row of timesheet data with its respective employees record, we need to perform ajoin on these fields. In SQLite, this is how this looks:
sqlite> SELECT first_name, last_name, date_worked, hours ...> FROM employees, timesheets ...> WHERE employees.id = timesheets.employee_id;first_name last_name date_worked hours
---------- ---------- ----------- -----
Alex Gladstone 20031229 4
Alex Gladstone 20031229 2
Alex Gladstone 20040102 6
Brenda Dudson 20031229 3
Brenda Dudson 20040102 5
Colin Aynsley 20031229 2
Colin Aynsley 20040102 3
Colin Aynsley 20031229 8
Debbie Churchill 20040102 8
Debbie Churchill 20040102 4
The repeated information is the reason we split these tables up in the first place. Rather than store each name and anyother employee details in the timesheets table, if we split off the employees data we only need to keep one copy.
In reality many more hours will be logged than in our example. So, supposing Brenda Dudson married and changedher name, we would have to update the last_name field for every row in timesheets. With the normalized relationshipbetween the tables, only one data item needs to be changed.
In the preceding example we had to use employees.id to qualify that the join was to the id field in the employeestabletimesheets also has a field named id. Using tablename.fieldname is the way to specify exactly which field you aretalking about. If this were left out, SQLite would throw an error rather than attempt to make the decision for you:
sqlite> SELECT first_name, last_name, date_worked, hours ...> FROM employees, timesheets ...> WHERE id = employee_id;SQL error: ambiguous column name: id
Notice that our selected columns were not qualified in this way. When the column name is unique across all the tablesin a query, this is fine. If we wanted to fetch both id fields, though, they would need qualifying the same way.
To save typing the table name in full repeatedly whenever a column name has to be qualified, you can specify a tablealias immediately after each table name in the FROM section. The following example aliases employees to e andtimesheets to t, and the single-letter aliases are then used to specify which id fields are referenced in the selectedcolumns list and WHERE clause.
sqlite> SELECT e.id, first_name, last_name, t.id, date_worked, hours ...> FROM employees e, timesheets t ...> WHERE e.id = t.employee_id;e.id first_name last_name t.id date_worked hours
---- ---------- ---------- ---- ----------- -----
101 Alex Gladstone 1 20031229 4
101 Alex Gladstone 2 20031229 2
101 Alex Gladstone 3 20040102 6
102 Brenda Dudson 4 20031229 3
102 Brenda Dudson 5 20040102 5
103 Colin Aynsley 6 20031229 2
103 Colin Aynsley 7 20040102 3
103 Colin Aynsley 8 20031229 8
104 Debbie Churchill 9 20040102 8
104 Debbie Churchill 10 20040102 4
Note
We did not specify any id values when the timesheet data was inserted; the INTEGER PRIMARY KEY attributehas assigned sequential values starting from 1.
A query can join many tables together, not just two. In the following example, we join employees, timesheets,projects, and clients to produce a report showing the title of all the projects and the names of all the employees thathave worked on each one.
sqlite> SELECT c.company_name, p.title, e.first_name, e.last_name ...> FROM clients c, projects p, timesheets t, employees e ...> WHERE c.id = p.client_id ...> AND t.project_code = p.code ...> AND e.id = t.employee_id;c.company_name p.title e.first_name e.last_name
Aggregate functions provide a quick way to find summary information about any of your data. The very simplestoperation is a count, which is executed as follows:
sqlite> SELECT count(*) FROM employees;count(*)
----------
4
The number of rows in the dataset is displayed. Of course this can be used in conjunction with a WHERE clause; forinstance, we can see how many male employees there are.
sqlite> SELECT count(*) FROM employees ...> WHERE sex = 'M';count(*)
----------
2
However, rather than executing one query for each possible value of sex, we can use a GROUP BY clause and anaggregate function to show us the number of employees of each sex.
sqlite> SELECT sex, count(*) ...> FROM employees ...> GROUP BY sex;sex count(*)
---------- ----------
F 2
M 2
Other aggregate functions available include sum(), avg(), min(), and max(), which find, respectively, the sum andaverage of a set of integers and the largest and smallest values from the set.
To find the average current hourly rate paid, we could use avg() as follows:
sqlite> SELECT avg(rate) FROM employee_rates ...> WHERE end_date IS NULL;avg(rate)
----------
23.75
To find the averages for men and women, we would include sex in the column list, and then use the GROUP BYclause on this column.
sqlite> SELECT sex, avg(rate) ...> FROM employees e, employee_rates r ...> WHERE e.id = r.employee_id ...> GROUP BY sex;sex avg(rate)
---------- ----------
F 20
M 27.5
We'll see in Part II of this book how you can create your own aggregating functions using the language APIs thatSQLite provides.
Ordering Data
Unless it is specified, the order in which selected rows are returned is undefinedthough they are usually brought backin the order in which they were inserted. SQLite uses the ORDER BY clause to specify the ordering of a dataset.
We actually inserted our employees in order of first name, but if we want the result of our SELECT statement to sorton last_name, we can add this in an ORDER BY clause:
sqlite> SELECT last_name, first_name ...> FROM employees ...> ORDER BY last_name;last_name first_name
---------- ----------
Aynsley Colin
Churchill Debbie
Dudson Brenda
Gladstone Alex
The ORDER BY can specify more than one column for sorting. Note that the columns listed need not actually appearin the list of columns selected. In the following example we order by sex first, then by last_name. The two womenappear at the top of the list in alphabetical order, followed by the men.
sqlite> SELECT last_name, first_name ...> FROM employees ...> ORDER BY sex, last_name;last_name first_name
---------- ----------
Churchill Debbie
Dudson Brenda
Aynsley Colin
Gladstone Alex
The ORDER BY clause is one situation where the data type of the columns referenced actually does matter, becauseSQLite needs to make a decision on how to evaluate a comparison between two values.
For instance if a column does not have a data type, what determines whether a value of 100 is higher or lower than avalue of 50? As two integers, clearly 100 is higher that 50, but a string comparison would consider 50 "higher"because its first character appears later in the ASCII character set.
In earlier versions of SQLite (version 2.6.3 and below), all comparisons for sorting were treated as numeric, withstrings being compared alphabetically only if they could not be evaluated as numbers. Since version 2.7.0, a columnwill take one of two types, either numeric or text, and the CREATE TABLE statement actually determines which typeis used.
Don't be confuseda numeric column type is still typeless inasmuch as it can store any kind of data, so it is notrestricted to integer, as an INTEGER PRIMARY KEY is, or indeed to any other numeric type. The fact that acolumn is considered numeric only comes into play when a value has to be compared.
You saw before that any number of names can make up a data type name in the CREATE TABLE statement. If thename contains one or more of the following strings, the column will be declared text. For all other data type names, orif the data type is omitted, it will be numeric.
BLOB
CHAR
CLOB
TEXT
To confirm the data type of a column you can use the typeof() function. Because the function would apply to theentire dataset, limit the records returned using LIMIT.
Expressions also have an implicit data type for comparisons, usually determined by the outermost operator regardlessof the actual arguments. An arithmetic operator returns a numeric result and a string operator returns a text result,regardless of the implied types of the arguments themselves.
As you might expect, you can also check the data type of an expression using typeof():
Although this problem does not occur with our demo database, sometimes a query will return more rows than youwant to process. You can use the LIMIT clause to limit the number of rows returned by specifying a number:
sqlite> SELECT * FROM employees LIMIT 2;id first_name last_name sex email
LIMIT can be used in conjunction with a WHERE clause to further restrict the dataset after any conditions have beenevaluated. Any ORDER BY clause specified is also taken into account first before limiting takes place. The followingexample fetches only the first record alphabetically from all the male employees.
sqlite> SELECT * FROM employees ...> WHERE sex = 'M' ...> ORDER by last_name, first_name ...> LIMIT 1;id first_name last_name sex email
LIMIT optionally takes an offset parameter, so that rather than returning the first N rows, it will skip a number ofrows from the top of the dataset and then return the next N rows. In the following example, rows 3 and 4 are returnedfrom the query, using an offset of 2 and a limit of 2.
sqlite> SELECT * FROM projects ...> ORDER BY due_date DESC ...> LIMIT 2, 2;code client_id title start_date due_date
To modify records in your database or remove them, use the SQL commands UPDATE and DELETE. Both use aWHERE clause to specify which rows are affected, so the syntax will be fairly familiar to you by now.
Suppose we want to split some of the hours worked on the Ongoing Consultancy project for ABC Enterprises offinto another project. The UPDATE statement would look something like this:
UPDATE timesheets
SET project_code = new_project_codeWHERE condition
Let's follow this example through and move all the hours spent in 2003 into a new project code ABC2003. First weinsert a new project record.
sqlite> INSERT INTO projects (code, client_id, title, start_date, due_date) ...> VALUES ('ABC2003', 502, 'Work in 2003', 20030101, 20031231);
Then we can run an UPDATE command to move all the 2003 hours to this new project.
sqlite> UPDATE timesheets ...> SET project_code = 'ABC2003' ...> WHERE project_code = 'ABCCONS' ...> AND date_worked BETWEEN 20030101 and 20031231;
Note
In this example we have assigned our own primary key value as project_code. If the insert had been to a table usingan INTEGER PRIMARY KEY, the primary key value could have been found using last_insert_rowid().
The UPDATE is silent unless there is a problem, so there is no confirmation on screen. The following query to find thetotal number of hours worked on each project and the extent of the work dates can be used to show that the updatehas indeed taken place.
sqlite> SELECT project_code, count(*), min(date_worked), max(date_worked) ...> FROM timesheets ...> GROUP BY project_code;project_code count(*) min(date_worked) max(date_worked)
You can update multiple columns in one UPDATE statement by supplying a comma-separated list of assignments.For instance, to update the contact name and telephone number for a client in one statement, you could do thefollowing:
sqlite> UPDATE clients ...> SET contact_name = 'Mr A. Newman', ...> telephone = '555-8888' ...> WHERE id = 501;
Let's say you change your mind about the previous reallocation of timesheets and want to remove project codeABC2003. The DELETE syntax for this reads as you might expect:
sqlite> DELETE FROM projects ...> WHERE code = 'ABC2003';
A list of columns is not required for a DELETE because action is taken on a record-by-record basis. Asking SQLiteto delete only a particular column makes no sense. To blank the values of specific columns while keeping the record inthe database, use an UPDATE and set the necessary fields to NULL.
Note
The WHERE clause in an UPDATE or DELETE statement is very important. If no WHERE clause is specified,action will be taken against every row in a tablewhich is usually not the desired result. Performing DELETE FROMtablename; will empty that table completely!
Left Joins
The DELETE we did in the preceding section has actually left our data in quite a mess. The relationship betweentimesheets and projects is broken because we now have timesheet data with a nonexistent project_code.
Some database systems have a cascading delete facility, which means that when a FOREIGN KEY is defined on atable, if that key value is deleted from the master table, the corresponding child records are also deleted. SQLite doesnot support this functionality in the same way, although we will look at a way it can be implemented with a trigger inthe next chapter.
However, now that we are in this situation, we can use it to demonstrate the LEFT JOIN operator in SQLite. Thejoins you saw previously used a WHERE clause to specify the relationship between the tables, for instancetimesheets.project_code = projects.code.
A LEFT JOIN also uses a condition to combine the data from both the tables, but where a row from the first tabledoes not have a corresponding row in the second, a record is still returnedcontaining NULL values for each of thefields in the second table.
Let's see this in action. The following query specifies a LEFT JOIN between timesheets and projects so that ourorphaned timesheet records still appear, but with blank project information.
sqlite> SELECT p.title, p.code, sum(t.hours) ...> FROM timesheets t ...> LEFT JOIN projects p ...> ON p.code = t.project_code ...> GROUP BY p.title, p.code ...> ORDER BY p.title;p.title p.code sum(t.hours)
------------------------ -------- ------------
4
Ongoing consultancy ABCCONS 4
Ordering system ACME01 23
Thing 3000 PREM3K 6
Thing 4000 PREM4K 8
So to find details of the timesheet entries that do not have a matching project code, we can modify the precedingquery and use an IS NULL condition to take advantage of this property of a LEFT JOIN.
sqlite> SELECT t.* ...> FROM timesheets t ...> LEFT JOIN projects p ...> ON p.code = t.project_code ...> WHERE p.code IS NULL;t.id t.employee_id t.project_code t.date_worked t.hours
There is another, often more readable, way to perform the same timesheet query using a nested subquery. SQLiteallows you to use the dataset produced by any query as the argument to an IN or NOT IN expression.
The following query finds all the timesheet records where the project code is not one of those in the projects table.
sqlite> SELECT * ...> FROM timesheets ...> WHERE project_code NOT IN ( ...> SELECT code ...> FROM projects ...> );id employee_id project_code date_worked hours
Though more readable, the NOT IN query in this example is usually less efficient than the LEFT JOIN method. Wewill deal with query optimization in more detail in Chapter 4, "Query Optimization."
Cartesian Joins
Another type of join possible in SQLite is the Cartesian join. This is something that usually only happens by mistake,so we mention it here only so that you are able to spot such problems.
If two (or more) tables are selected without a proper join in the WHERE clause, every row in table 1 is paired witheach row in table 2 to produce a Cartesian productthe number of rows returned is the product of the number of rowsin each table.
The following query shows a Cartesian join on the clients and projects tables, and you should be able to see that withtables containing significant amounts of data, a Cartesian join is generally not a desirable result.
Transactions are the way that databases implement atomicitythe assurance that the entirety of a request to alter thedatabase is acted upon and not just part of it. Atomicity is one of the key requirements of a mission-critical databasesystem.
Every time the database is changedin other words for any INSERT, UPDATE, or DELETE to take placethe changemust take place inside a transaction. SQLite will start an implicit transaction whenever one of these commands isissued if there is not already a transaction in progress.
You would want to start a transaction explicitly if you wanted to make a series of changes that must all take place atthe same time. Simply issuing the commands one after each other does not absolutely guarantee that if the earliercommands are executed the later commands will also have been processed.
Let's say in our demo database that we wanted to change one of the project codes, and because the timesheets tableis joined to projects on project_code, we need to ensure that the same change is made to both tables at the sametime. In SQLite we do this as follows:
sqlite> BEGIN TRANSACTION;sqlite> UPDATE projects ...> SET code = 'ACMENEW' ...> WHERE code = 'ACME01';sqlite> UPDATE timesheets ...> SET project_code = 'ACMENEW' ...> WHERE project_code = 'ACME01';sqlite> COMMIT TRANSACTION;
Until the COMMIT TRANSACTION command is issued, neither of the UPDATE statements have caused a changeto be made to the database. A query executed within the transaction will consider these changes to have been made;however, the ROLLBACK TRANSACTION command can be used to cancel the entire transaction. The followingexample shows a mistake made when making an update that was rolled back:
sqlite> BEGIN TRANSACTION;sqlite> UPDATE clients ...> SET contact_name = 'Mr S. Error';sqlite> SELECT * from clients;id company_name contact_name telephone
504 Integer Primary Key Corp Mr A. Increment 555-1234
The sqlite_master Table
You've already seen that from the sqlite tool you can view the list of tables in a database with the .tables commandand view the schema of any table using .schema.
However, SQLite is designed as an embedded database, and ultimately you will be writing applications in yourlanguage of choice and calling SQLite API functions where the commands beginning with a period that are available insqlite cannot be invoked directly.
We finish this chapter by looking at the internal tables that can be used to query a database and its table schemas.
The master table holding the key information about your database tables is called sqlite_master, and using sqlite youcan see its schema as you might expect:
The type field tells us what kind of object we are dealing with. So far we've only looked at tables, but indexes andtriggers are also stored in sqlite_master, and we'll look at these in more detail in Chapter 3.
For a table, name and tbl_name are the same. An index or trigger will have its own identifier, but tbl_name willcontain the name of the table to which it applies. The sql field contains the CREATE TABLE (or CREATE INDEXor CREATE TRIGGER) statement issued when each database object was created.
Note
The sqlite_master table is read-only. You cannot modify it directly using UPDATE, DELETE, or INSERTstatements.
So, to find the names of all tables in a database, do the following:
sqlite> SELECT tbl_name ...> FROM sqlite_master ...> WHERE type = 'table';tbl_name
--------------------
employees
employee_rates
clients
projects
timesheets
Or to find the SQL command used to create a table, the following query will workshown here in line output mode sothat you can see the whole text.
sqlite> SELECT sql ...> FROM sqlite_master ...> WHERE type = 'table' ...> AND tbl_name = 'projects'; sql = CREATE TABLE projects (
Chapter 3. SQLite Syntax and Use In this chapter we look in detail at the SQL syntax understood by SQLite. We will discuss the full capabilities of thelanguage and you will learn to write effective, accurate SQL.
You have already come across most of the supported SQL commands in Chapter 2, "Working with Data," in thecontext of the demo database. This chapter builds on that knowledge by exploring the syntax and usage of eachcommand in more detail to give a very broad overview of what you can do using SQLite.
Naming Conventions Each database, table, column, index, trigger, or view has a name by which it is identified and almost always the nameis supplied by the developer. The rules governing how a valid identifier is formed in SQLite are set out in the next fewsections.
Valid Characters
An identifier name must begin with a letter or the underscore character, which may be followed by a number ofalphanumeric characters or underscores. No other characters may be present. These identifier names are valid:
mytable
my_field
xyz123
a
However, the following are not valid identifiers:
my table
my-field
123xyz
You can use other characters in identifiers if they are enclosed in double quotes (or square brackets), for example:
SQLite does not have a fixed upper limit on the length of an identifier name, so any name that you find manageable towork with is suitable.
Reserved Keywords
Care must be taken when using SQLite keywords as identifier names. As a general rule of thumb you should try toavoid using any keywords from the SQL language as identifiers, although if you really want to do so, they can be usedproviding they are enclosed in square brackets.
For instance the following statement will work just fine, but this should not be mimicked on a real database for thesake of your own sanity.
For the most part, case sensitivity in SQLite is off. Table names and column names can be typed in uppercase,lowercase, or mixed case, and different capitalizations of the same database object name can be usedinterchangeably.
SQL commands are always shown in this book with the keywords in uppercase for clarity; however, this is not arequirement.
Note
The CREATE TABLE, CREATE VIEW, CREATE INDEX, and CREATE TRIGGER statements all store theexact way in which they were typed to the database so that the command used to create a database object can beretrieved by querying the sqlite_master table. Therefore it is always a good idea to format your CREATE statementsclearly, so they can be referred to easily in the future.
Creating and Dropping Tables Creating and dropping database tables in SQLite is performed with the CREATE TABLE and DROP TABLEcommands respectively. The basic syntax for CREATE TABLE is as follows:
Simply put, a table may be declared as temporary, if desired, and the structure of each table has to have one or morecolumn definitions followed by zero or more constraints.
Table Column Definitions
A column definition is defined as follows:
name [type] [[CONSTRAINT name] column-constraint]*
As you saw in Chapter 2, SQLite is typeless and therefore the type attribute is actually optional. Except for anINTEGER PRIMARY KEY column, the data type is only used to determine whether values stored in that column areto be treated as strings or numbers when compared to other values.
You can use the optional CONSTRAINT clause to specify one or more of the following column constraints thatshould be enforced when data is inserted:
NOT NULL
DEFAULT
PRIMARY KEY
UNIQUE
A column declared as NOT NULL must contain a value; otherwise, an INSERT attempt will fail, as demonstrated inthe following example:
sqlite> CREATE TABLE vegetables ( ...> name CHAR NOT NULL, ...> color CHAR NOT NULL ...> );
sqlite> INSERT INTO vegetables (name) VALUES ('potato');SQL error: vegetables.color may not be NULL
Often, a column declared NOT NULL is also given a DEFAULT value, which will be used automatically if thatcolumn is not specified in an INSERT. The following example shows this in action.
sqlite> CREATE TABLE vegetables ( ...> name CHAR NOT NULL, ...> color CHAR NOT NULL DEFAULT 'green' ...> );
sqlite> INSERT INTO vegetables (name, color) VALUES ('carrot', 'orange');sqlite> INSERT INTO vegetables (name) VALUES ('bean');sqlite> SELECT * FROM vegetables;name color
---------- ----------
carrot orange
bean green
However, if you attempt to insert NULL explicitly into a NOT NULL column, SQLite will still give an error:
sqlite> INSERT INTO vegetables (name, color) VALUES ('cabbage', NULL);SQL error: vegetables.color may not be NULL
Functionally, a PRIMARY KEY column behaves just the same as one with a UNIQUE constraint. Both types ofconstraint enforce that the same value may only be stored in that column once, but other than the special case of anINTEGER PRIMARY KEY, the only point to note is that a table can have only one PRIMARY KEY column.
SQLite will raise an error whenever an attempt is made to insert a duplicate value into a UNIQUE or PRIMARYKEY column, as shown in the following example. This example also shows that a column can be declared as bothNOT NULL and UNIQUE.
sqlite> CREATE TABLE vegetables ( ...> name CHAR NOT NULL UNIQUE, ...> color CHAR NOT NULL ...> );sqlite> INSERT INTO vegetables (name, color) VALUES ('pepper', 'red');sqlite> INSERT INTO vegetables (name, color) VALUES ('pepper', 'green');SQL error: column name is not unique
Resolving Conflicts
NOT NULL, PRIMARY KEY, and UNIQUE constraints may all be used in conjunction with an ON CONFLICTclause to specify the way a conflict should be resolved if an attempt to insert or modify data violates a columnconstraint.
The conflict resolution algorithms supported are
ROLLBACK
ABORT
FAIL
IGNORE
REPLACE
You could apply a constraint to the vegetables table from the preceding example as follows:
sqlite> CREATE TABLE vegetables ( ...> name CHAR NOT NULL UNIQUE ON CONFLICT REPLACE, ...> color CHAR NOT NULL ...> );
This time, because REPLACE was specified as the conflict resolution algorithm, inserting the same vegetable nametwice does not cause an error. Instead the new record replaces the conflicting record.
sqlite> INSERT INTO vegetables (name, color) VALUES ('pepper', 'red');sqlite> INSERT INTO vegetables (name, color) VALUES ('pepper', 'green');sqlite> SELECT * FROM vegetables;name color
---------- ----------
pepper green
The REPLACE algorithm ensures that an SQL statement is always executed, even if a UNIQUE constraint wouldotherwise be violated. Before the UPDATE or INSERT takes place, any pre-existing rows that would cause theviolation are removed. If a NOT NULL constraint is violated and there is no DEFAULT value, the ABORT algorithmis used instead.
The ROLLBACK algorithm causes an immediate ROLLBACK TRANSACTION to be issued as soon as theconflict occurs and the command will exit with an error.
When you use the ABORT algorithm, no ROLLBACK TRANSACTION is issued, so if the violation occurs withina transaction consisting of more than one INSERT or UPDATE, the database changes from the previous statementswill remain. Any changes attempted by the statement causing the violation, however, will not take place. For a singlecommand using only an implicit transaction, the behavior is identical to ROLLBACK.
The FAIL algorithm causes SQLite to stop with an error when a constraint is violated; however, any changes madeas part of that command up to the point of failure will be preserved. For instance, when an UPDATE statementperforms a change sequentially on many rows of the database, any rows affected before the constraint was violatedwill remain updated.
SQLite will never stop with an error when the IGNORE algorithm is specified and the constraint violation is simplypassed by. In the case of an UPDATE affecting multiple rows, the modification will take place for every row otherthan the one that causes the conflict, both before and after.
The ON CONFLICT clause in a CREATE TABLE statement has the lowest precedence of all the places in which itcan be specified. An overriding conflict resolution algorithm can be specified in the ON CONFLICT clause of aBEGIN TRANSACTION command, which can in turn be overridden by the OR clause of a COPY, INSERT, orUPDATE statement. We will see the respective syntaxes for these clauses later in this chapter.
The CHECK Clause
The CREATE TABLE syntax also allows for a CHECK clause to be defined, with an expression in parentheses.This is a feature included for SQL compatibility and is reserved for future use, but at the time of this writing is notimplemented.
Using Temporary Tables
Using CREATE TEMPORARY TABLE creates a table object in SQLite that can be queried and manipulatedexactly the same as a nontemporary table. However, the table will only be visible to the process in which it wascreated and will be destroyed as soon as the database is closed.
sqlite> INSERT INTO temptable (myfield) VALUES ('xyz');SQL error: no such table: temptable
The data inserted into a temporary table and its schema are not written to the connected database file, nor is there arecord created in sqlite_master. Instead a separate table, sqlite_temp_master, is used to reference temporary tables.
sqlite> CREATE TEMPORARY TABLE temptable ( ...> myfield char ...> );sqlite> SELECT * FROM sqlite_temp_master; type = table
name = temptable
tbl_name = temptable
rootpage = 3
sql = CREATE TEMPORARY TABLE temptable (
myfield char
)
Temporary tables are specific to the sqlite handle, not the process. Surprisingly, people often become confused aboutthis, particularly in Windows, where a common design pattern is to open a separate sqlite handle to the samedatabase from each thread.
Anatomy of a SELECT Statement The syntax definition for an SQL statement is
SELECT [ALL | DISTINCT] result [FROM table-list][WHERE expr][GROUP BY expr-list][HAVING expr][compound-op select]*[ORDER BY sort-expr-list][LIMIT integer [(OFFSET|,) integer]]
The only required item in a SELECT statement is the result, which can be one of the following:
The * character
A comma-separated list of one or more column names
An expression
The latter two bullet points should be combined into: "A comma-separated list of one or more expressions." Theoriginal two points make it seem as if the following would be an error:
SELECT a+1, b+1 FROM ab;
but this would be okay:
SELECT a, b FROM ab;
In fact, both are valid.
Using the * character or a list of columns makes no sense without a FROM clause, but in fact an expression whosearguments are constants rather than database items can be used alone in a SELECT statement, as in the followingexamples.
sqlite> SELECT (60 * 60 * 24);86400
sqlite> SELECT max(5, 20, -4, 8.7);20
sqlite> SELECT random();220860261
If the FROM list is omitted, SQLite effectively evaluates the expression against a table that always contains a singlerow.
The FROM list includes one or more table names in a comma-separated list, each with an optional alias name thatcan be used to qualify individual column names in the result. Where aliases are not used, the table name in full can beused to qualify columns.
For instance, the two following SELECT statements are identicalthe latter uses a table alias m for mytable.
SELECT mytable.myfield
FROM myfield;
SELECT m.myfield
FROM mytable m;
The WHERE Clause
The WHERE clause specifies one or more conditions used to impose restrictions on the dataset returned by aSELECT. It is used both to limit the number of rows returned and to indicate a relationship used to join two tablestogether.
The general usage to impose a condition on the rows in a table is as follows:
SELECT resultFROM table-listWHERE expr;
Expression expr generally involves a comparison of some kind on a table column, as shown in the following example:
SELECT *
FROM mytable
WHERE myfield = 'somevalue';
Table 3.1 shows the relational operators that can be used in a WHERE clause.
Table 3.1. Relational Operators
Operator Meaning
a = b a is equal to b
a != b a is not equal to b
a < b a is less than b
a > b a is greater than b
a <= b a is less than or equal to b
a >= b a is greater than or equal to b
a IN (b, c) a is equal to either b or c
a NOT IN (b, c) a is equal to neither b nor c
When you perform a comparison using a relational operatorand particularly the greater-than and less-than operators,< and >the data type of the column comes into play.
The following example shows how comparisons between the numbers 8, 11, and 101 differ greatly when performedas string operations. As integers, the order is as you would expect, however as strings, '101' is less than '11', which isin turn less than '8'. The individual character values in the string are compared from left to right in turn to determinewhich is the greatest value.
sqlite> CREATE TABLE compare (string TEXT, number INTEGER);sqlite> INSERT INTO compare (string, number) values ('101', 101);sqlite> SELECT number FROM compare WHERE number > 11;101
sqlite> SELECT string FROM compare WHERE string > '11';8
However, note that if you use a relational operator with a number argument that is not contained in quotes, an integercomparison is performed regardless of the column data type.
sqlite> SELECT string FROM compare WHERE string > 11;101
Selecting from multiple tables without a WHERE clause produces a Cartesian product of the datasets and is usuallynot a desirable result. With two tables, each record in table1 is paired with each record in table2. The total number ofrows returned is the product of the number of rows in each table in the FROM list.
The following example shows the result of a Cartesian product of three tables, each containing just two rows. In total,eight rows are returned (2x2x2).
In this example each table has a field called myfield, so each column in the result has to be qualified with theappropriate table name. This is not necessary where a column name is unique across all tables in the FROM list;however, it is good practice to always qualify column names to avoid ambiguity. If a column name could refer to morethan one table, SQLite will not make the decision. Instead an error is raised as shown in the following example:
To join two tables on a common fieldknown as an equi-join because the relationship is an equalitythe general syntax is
SELECT resultFROM table1, table2
WHERE table1.keyfield1 = table2.keyfield2
SQLite supports outer joins via the LEFT JOIN keyword, whereby each row in the left tablethe one specified first inthe SELECT statementis combined with a row from the right table. Where the join condition does not produce amatch between the two tables, rows from the left table are still returned but with NULL values for each column thatshould be in the right table.
The general syntax for a LEFT JOIN is as follows:
SELECT result
FROM table1
LEFT [OUTER] JOIN table2
ON table1.keyfield1 = table2.keyfield2
[WHERE expr]
The LEFT JOIN operator can be written as LEFT OUTER JOIN as a matter of preference; the OUTER keyword isoptional.
GROUP BY and Aggregate Functions
The GROUP BY clause is used to aggregate data into a single row where the value of one or more specified columnsis repeated. This feature can be used to reduce the number of records to only find unique values of a column, but isparticularly useful when used in conjunction with the SQLite's aggregate functions.
The GROUP BY clause takes a list of expressionsusually column names from the resultand aggregates data for eachexpression. In the vegetables table we created previously we had more than one green vegetable, so grouping on thecolor column will return each value only once.
sqlite> SELECT color ...> FROM vegetables ...> GROUP BY color;color
----------
green
orange
More interesting is to use the aggregate function count() to show how many records there are for each value of color:
sqlite> SELECT color, count(color) ...> FROM vegetables ...> GROUP BY color;color count(color)
---------- ------------
green 2
orange 1
Using count(fieldname) will return the number of rows containing a non-NULL value in that field. If you want to returna count of the total number of rows, regardless of any NULL values, count(*) will do this, as the following exampleshows:
Table 3.2 lists all the aggregate functions available in SQLite.
Table 3.2. Aggregate Functions
Function Meaning
avg(column) Returns the mean average of all values in column
count(column) Returns the number of times that a non-NULL valueappears in column
count(*) Returns the total number of rows in a query, regardlessof NULL values
max(column) Returns the highest of all values in column, using the usualsort order
min(column) Returns the lowest of all values in column, using the usualsort order
sum(column) Returns the numeric sum of all values in column
HAVING Clause
The HAVING clause is a further condition applied after aggregation takes place. In contrast to a WHERE clause,which applies a condition to individual elements in a table, HAVING is used to restrict records based on the summaryvalue of a grouping.
To return only rows from the vegetables table where there is more than one of the same color, we can do this:
sqlite> SELECT color, count(*) ...> FROM vegetables GROUP BY color ...> HAVING count(*) > 1;color count(*)
---------- ----------
green 2
It is actually not necessary for count(*) to appear in the result, as shown in the following example:
sqlite> SELECT color ...> FROM vegetables GROUP BY color ...> HAVING count(*) > 1;color
----------
green
Column Aliases
The column headings displayed in the output of a SELECT statement are usually the same as the items specified inthe result section. For a straight column, the name of the column is displayed. For an expression, however, theexpression text is used.
Although the column headings are only displayed in sqlite when .headers is set to on, it is important to know whateach column's name is so that all the columns can be referenced correctly from within a programming API. A columnalias is specified with the AS keyword to explicitly give a new name to a selected column.
In the following example, we give a column alias to the result of the count(*) aggregate function. In addition torenaming the column header, we can use the alias name in the HAVING clause, which can sometimes aid readabilityof code.
sqlite> SELECT color, count(*) AS num_colors ...> FROM vegetables GROUP BY color ...> HAVING num_colors > 1;color num_colors
---------- ----------
green 2
Attaching to Another Database Using sqlite, the .databases command lists all the databases that are open for the current session. There will alwaysbe two databases open after you invoke sqlitemain, the database specified on the command line, and temp, thedatabase used for temporary tables.
sqlite> .databases0 main /home/chris/sqlite/demodb
1 temp /var/tmp/sqlite VGazbfyWvuUr29P
It is possible to attach more databases to your current session with the ATTACH DATABASE statement. This addsa connection to another database without replacing your currently selected database.
The syntax is
ATTACH [DATABASE] database-filename AS database-name
The keyword DATABASE is optional and is used only for readability, but you must provide a unique database-nameparameter that will be used to qualify table references, essential in case more than one database could have the sametable name.
Suppose you are working on a new database called newdb and want to access some of the databases from ourdemo database from Chapter 2. The following example shows demodb being attached to the current sqlite session:
$ sqlite newdbSQLite version 2.8.12
Enter ".help" for instructions
sqlite> ATTACH DATABASE demodb AS demodb;sqlite> .databases0 main /home/chris/sqlite/newdb
1 temp /var/tmp/sqlite_VGazbfyWvuUr29P
2 demodb /home/chris/sqlite/demodb
Accessing tables from an attached database is straightforwardjust prefix any table name with the database name (thename given after the keyword AS, not the filename, if they are different) and a period.
We can perform a query on the clients table from demodb as follows:
sqlite> SELECT company_name FROM demodb.clients;company_name
--------------------
Acme Products
ABC Enterprises
Premier Things Ltd
Tables in the main database can be accessed using their table name alone, or qualified as main.tablename. If a tablename is unique across all databases attached in a particular session, it does not need to be prefixed with its databasename even if it is not in the main database. However, it is still good practice to qualify all tables when you are workingwith multiple databases to avoid confusion.
Note
The SQL commands INSERT, UPDATE, SELECT, and DELETE can all be performed on an attached database byusing the database name prefix. However, CREATE TABLE and DROP TABLE can only take place on the maindatabaseyou must exit sqlite and begin a new session if you want to manipulate tables from a different database.
Note the situation with multi-database transactions here. If a machine or software failure occurs, a transaction is onlyatomic within one database. If more than one database were written to within a single transaction, one database mightbe committed and the other rolled back in the event of a failure.
There is a compile-time limit of 10 attached database files by default. This can be increased to up to 255 concurrentdatabases by modifying the following line in src/sqliteInt.h:
#define MAX_ATTACHED 10
To detach an attached database, the syntax is simply
DETACH [DATABASE] database-name
Manipulating Data Next we'll look at how records can be added to a database and demonstrate different ways of using the INSERTcommand, and examine the syntax of the SQL UPDATE and DELETE commands.
Transactions
Any change to a SQLite database must take place within a transactiona block of one or more statements that alterthe database in some way. Transactions are the way in which a robust database system ensures that either all or noneof the requests to alter the database is carried out; it can never be just partially completed. This property of adatabase is called atomicity.
Whenever an INSERT, UPDATE, or DELETE command is issued, SQLite will begin a new transaction unless onehas already been started. An implicit transaction lasts only for the duration of the one statement but ensures that, forinstance, an UPDATE affecting many rows of a large table will always carry out the action on every row orin theunlikely event of a system failure while processing this commandnone of them. The database will not reflect a changeto any row until every row has been updated and the transaction closed.
A transaction can be started from SQL if you want to make a series of changes to the database as one atomic unit.This is the syntax of the BEGIN TRANSACTION statement:
BEGIN [TRANSACTION [name]] [ON CONFLICT conflict-algorithm]
The transaction name is optional and, currently, is ignored by SQLite. The facility to provide a transaction name isincluded for future use if the ability to nest transactions is added. Currently only one transaction can be open at a time.In fact the keyword trANSACTION is also optional, but is included for readability.
An ON CONFLICT clause can be specified to override the default conflict resolution algorithm specified at the tablelevel, but can be superseded itself by the OR clause of an INSERT, UPDATE, or DELETE statement.
To end a transaction and save changes to the database, use COMMIT TRANSACTION. The optional transactionname may be specified. To abort a transaction without any of the changes being stored, use ROLLBACKTRANSACTION.
Inserting Data
There are two versions of the syntax for the INSERT statement, depending on where the data to be inserted iscoming from.
The first syntax is the one we have already used in Chapter 2, to insert a single row from values provided in thestatement itself. The second version is used to insert a dataset returned as the result of a SELECT statement.
INSERT Using VALUES
The syntax for a single-row insert using the VALUES keyword and a list of values provided as part of the statementis as follows:
INSERT [OR conflict-algorithm]
INTO [database-name .] table-name [(column-list)]
VALUES (value-list)
Although all our examples so far have included a column-list, it is actually optional. Where no column-list is provided,the value-list is assumed to contain one value for each column in the table, in the order they appear in the schema.
This can be a useful shortcut when you are adding data; for instance because we know the column-list is a name andthen a color, a record can be inserted into the vegetables table simply using this format:
sqlite> INSERT INTO vegetables VALUES ('mushroom', 'white');
However if the schema of the table is not what you are expecting, the INSERT will fail with an error. SQLite wouldnot make any assumption as to which columns you are referring to.
Because SQLite does not have an ALTER TABLE command, it is much harder to change a schema after a table hasbeen created than it is with other database engines that include this command. We'll see a workaround for ALTERTABLE in the following example, and if for any reason the vegetables table had been expanded to include threecolumns, the same INSERT statement would produce this error:
sqlite> INSERT INTO vegetables VALUES ('mushroom', 'white');SQL error: table vegetables has 3 columns but 2 values were supplied
Therefore it is good practice to always include the column-list in an INSERT statementalso known as performing a fullinsert.
The OR keyword is used to specify a conflict resolution algorithm in the same way we saw for the CREATE TABLEstatement. The list of algorithms and their behavior is identical, but the keyword OR is used instead of ONCONFLICT to give a more natural-sounding syntax.
The conflict algorithm in the OR clause of an INSERT statement has the highest precedence possible, and willoverride any other setting present at the table or transaction level.
INSERT Using SELECT
The syntax to insert the result of a SELECT query into another table is as follows:
INSERT [OR conflict-algorithm]
INTO [database-name .] table-name [(column-list)] select-statement
The select-statement should return a dataset with the same number and order as the columns specified in thecolumn-list (or every column in the destination table if no column-list is supplied). The full syntax of the SELECTstatement is available, and any number of rows can be returned.
As with INSERT ... VALUES, the column-list is optional but including it is highly advisable.
Updating Data
The syntax of the UPDATE statement in SQLite is as follows:
One or more assignments can be performed within the same statement upon the same subset of data, defined by theoptional WHERE clause. An assignment is defined as
column-name = expr
Although the WHERE clause is not required, it is usually desirable. The following example would assign the value ofcolor to green for every row in the table, when in fact we probably only meant to update one or a few records.
sqlite> UPDATE vegetables ...> SET color = 'green';
The WHERE clause can be as simple or complex as necessary, and all the conditional elements that can be used inthe WHERE clause of a SELECT statement can be used here.
It is not logical to join two or more tables when performing an UPDATE; however, subselects can be used in theWHERE clause, as in the following example:
sqlite> UPDATE mytable ...> SET myfield = 'somevalue' ...> WHERE mykey IN ( ...> SELECT keyfield ...> FROM anothertable ...> );
The OR keyword is used in an UPDATE statement to specify a conflict resolution algorithm with the highestprecedence possible, in the same way as with an INSERT.
Deleting Data
The DELETE statement is used to remove rows from a database. Its syntax is
DELETE FROM [database-name .] table-name [WHERE expr]
As with the UPDATE statement, the WHERE clause is optional but is usually desiredperforming a DELETE on atable with no WHERE clause will empty the table. No column-list is required for a DELETE because the operationaffects the entire row.
The WHERE clause can use the AND and OR operators to combine conditions and can use subselects to perform aDELETE operation conditional on the results of another query.
The following example modifies a query from Chapter 2 to remove records from the timesheets table where theproject_code field does not correspond to a key in the projects table.
sqlite> DELETE FROM timesheets ...> WHERE project_code NOT IN ( ...> SELECT code ...> FROM projects ...> );
Altering a Table Schema
There is no ALTER TABLE statement in SQLite; instead a table must be dropped and re-created with a new fieldadded, with any data that you want to preserve extracted before the table is dropped and reloaded into the newstructure.
A temporary table is the ideal place to hold such data, and the CREATE TABLE ... AS syntax gives us a very easyway to create a copy of an existing table. The syntax is simply
CREATE [TEMP | TEMPORARY TABLE] table-name AS select-statement
Let's suppose we want to add a new descriptive column to our vegetables table but without losing the data we havealready created. The first step is to take a copy of the existing vegetables table to a new temporary table.
sqlite> CREATE TEMPORARY TABLE veg_temp ...> AS SELECT * FROM vegetables;
However, only the field specification has been copied when a table is created this way. The schema of the new tabledoes not include any data type names or column constraints. It is not possible to give a set of column definitions whenusing CREATE TABLE ... AS in SQLite.
So now we can safely drop the old vegetables table and re-create it with our new field:
sqlite> DROP TABLE vegetables;sqlite> CREATE TABLE vegetables ( ...> name CHAR NOT NULL, ...> color CHAR NOT NULL DEFAULT 'green', ...> description CHAR ...> );
Finally, reinstate the copied data from the temporary table using the INSERT ... SELECT syntax:
sqlite> INSERT INTO vegetables (name, color) ...> SELECT name, color FROM veg_temp;
Loading Data from a File
The COPY command in SQLite was based on a similar command found in PostgreSQL and as a result is designedto read the output of the pg_dump command to facilitate data transfer between the two systems.
However, COPY can also be used to load data from most delimited text file formats into SQLite. It has the followingsyntax:
COPY [OR conflict-algorithm] [database-name .] table-name FROM filename
[USING DELIMITERS delim]
The destination table table-name must existthough it need not be emptybefore the COPY operation is attempted, andeither filename must be in the current directory or a full path given.
Each line in the input file will become a record in the table, with each column separated by a tab character, unless adifferent delimiter character is specified in the USING DELIMITERS clause.
If a tabor the specified delimiterappears within a data column, it must be escaped with a backslash character. Thebackslash itself can appear in the data if it is escaped itself; in other words it will appear as two consecutive backslashcharacters.
The special character sequence \N in the data file can be used to represent a NULL value.
Tab is used to separate columns in the output of pg_dump, so it is the default delimiter for the COPY command.Another popular format is comma-separated values (CSV). Listing 3.1 shows a comma-separated data file that canbe loaded into the three-column vegetables table.
Listing 3.1. vegetables.csv
cucumber,green,Long green salad vegetable
pumpkin,orange,Great for Halloween
avocado,green,Can't make guacamole without it
The COPY command to load this data file into SQLite is
sqlite> COPY vegetables FROM 'vegetables.csv' ...> USING DELIMITERS ',';
You can instruct COPY to read data from the standard input stream instead of a file by using the keyword STDINinstead of a filename. A blank line, or a backslash followed by a period, is used to indicate the end of the input.
COPY permits an overriding conflict resolution algorithm to be specified after the OR keyword, as with INSERTand UPDATE.
Indexes The subject of keys and indexes and how they can affect the performance of your database will be addressed in Chapter 4, "Query Optimization," but first we will examine the syntax for creating and finding information on tableindexes.
Creating and Dropping Indexes
The CREATE INDEX command is used to add a new index to a database table, using this syntax:
CREATE [UNIQUE] INDEX index-name
ON [database-name .] table-name (column-name [, column-name]*)
[ON CONFLICT conflict-algorithm]
The index-name is a user-provided identifier for the new index and must be unique across all database objects. Itcannot take the same name as a table, view, or trigger. A popular naming convention is to use the table name and thecolumn name(s) used for the index key separated by an underscore character.
To add an index to the color column of the vegetables table, we would use the following command.
sqlite> CREATE INDEX vegetables_color ...> ON vegetables(color);
The syntax of column-name allows for a sort order to be given after each column name, either ASC or DESC;however, currently in SQLite this is ignored. At the present time, all indexes are created in ascending order.
Removing an index is done with reference to the identifier given when it was created, which you can always find byquerying the sqlite_master table if you cannot remember it.
sqlite> SELECT * FROM sqlite_master ...> WHERE type = 'index'; type = index
name = vegetables_color
tbl_name = vegetables
rootpage = 10
sql = CREATE INDEX vegetables_color
ON vegetables(color)
The DROP INDEX command works as you might expect:
sqlite> DROP INDEX vegetables_color;
Don't worry if you misread the sqlite_master output and use the table name instead of the index name. SQLite onlyallows you to drop indexes with the DROP INDEX command and tables with the DROP TABLE command.
sqlite> DROP INDEX vegetables;SQL error: no such index: vegetables
UNIQUE Indexes
The UNIQUE keyword is used to specify that every value in an indexed column is unique. Where an index is createdon more than one column, every permutation of the column values has to be unique, even though the same value mayappear more than once in its own column.
Since we have already inserted several vegetables of the same color into the table, SQLite will give an error if weattempt to create a unique index on the color field.
sqlite> CREATE UNIQUE INDEX vegetables_color ...> ON vegetables(color);SQL error: indexed columns are not unique
The ON CONFLICT clause at the index level is only relevant for a UNIQUE index; otherwise, there will never be aconflict on the data it applies to. The conflict resolution algorithm is used when an INSERT, UPDATE, or COPYstatement would cause the unique constraint of the index to be violated. It cannot be used in the preceding CREATEUNIQUE INDEX statement to force a unique index onto a column containing multiple values.
The default conflict resolution algorithm is ABORT, and the same list of algorithms is permitted for indexes as in theCREATE TABLE statement.
Views A view is a convenient way of packaging a query into an object that can itself be used in the FROM clause of aSELECT statement.
Creating and Dropping Views
The syntax for CREATE VIEW is shown next.
CREATE [TEMP | TEMPORARY] VIEW view-name AS select-statement
The select-statement can be as simple or as complex as necessary; it could return the subset of a single table based ona conditional WHERE clause, or join many tables together to form a single object that can be more easily referencedin SQL.
To drop a view, simply use the DROP VIEW statement with the view-name given when it was created.
A view is not a table. You cannot perform an UPDATE, INSERT, COPY, or DELETE on a view, but if the data inone of the source tables changes, those changes are reflected instantly in the view.
Using Views
The following example shows a view based on the demo database tables employees and employee_rates using aquery that returns the current rate of pay for each employee.
sqlite> CREATE VIEW current_pay AS ...> SELECT e.*, er.rate ...> FROM employees e, employee_rates er ...> WHERE e.id = er.employee_id ...> AND er.end_date IS NULL;
We can then query the new view directly, even adding a new condition in the process:
sqlite> SELECT * FROM current_pay ...> WHERE sex = 'M';id first_name last_name sex email rate
The column names in a view are the column names from the table. Where an expression is used, SQLite will faithfullyreproduce the expression as the column heading.
sqlite> CREATE VIEW veg_upper AS ...> SELECT upper(name), upper(color) ...> FROM vegetables;sqlite> SELECT * FROM veg_upper LIMIT 1;upper(name)|upper(color)
CARROT|GREEN
However, the column in the view cannot actually be called upper(name). As shown in the following example, SQLitewill attempt to evaluate the upper() function on the nonexistent name column.
sqlite> SELECT upper(name) from veg_upper;SQL error: no such column: name
Column aliases can be used to give an explicit name to a column so that they can be referenced within a subsequentquery.
sqlite> CREATE VIEW veg_upper AS ...> SELECT upper(name) AS uppername, upper(color) AS uppercolor ...> FROM vegetables;sqlite> SELECT * FROM veg_upper ...> WHERE uppercolor = 'ORANGE';uppername|uppercolor
CARROT|ORANGE
PUMPKIN|ORANGE
Note
When a view includes two columns with the same namewhether it is the same column selected twice from one table,or once each from two tables that happen to share a column nameSQLite will modify the column names in the viewunless aliases are used. A duplicate column will be suffixed with _1 the first time it appears, _2 the second time, andso on.
SQLite does not validate the select-statement SQL in CREATE VIEW. You will only know if there is an error in theSELECT when you come to query the new view. The view's SELECT statement is effectively substituted into thequery at the point where view-name appears, so the errors displayed may not appear to reflect the query you typed.
The following example creates a view with a deliberate errorthere is no column entitled shape in vegetablesand showsthat the error is not detected until you query the view.
sqlite> CREATE VIEW veg_error AS ...> SELECT shape FROM vegetables;sqlite> SELECT * from veg_error;SQL error: no such column: shape
Triggers A trigger is an event-driven rule on a database, where an operation is initiated when some other transaction (event)takes place. Triggers may be set to fire on any DELETE, INSERT, or UPDATE on a particular table, or on anUPDATE OF particular columns within a table.
Creating and Dropping Triggers
The syntax to create a trigger on a table is as follows:
CREATE [TEMP | TEMPORARY] TRIGGER trigger-name
[BEFORE | AFTER] database-event ON [database-name .]table-name
trigger-action
The TRigger-name is user-specified and must be unique across all objects in the databaseit cannot share the samename as a table, view, or index.
The trigger can be set to fire either BEFORE or AFTER database-event; that is, either to pre-empt the transactionand perform its action just before the UPDATE, INSERT, or DELETE takes place, or to wait until the operation hascompleted and then immediately carry out the required action.
If the database-event is specified as UPDATE OF column-list, it will create a trigger that will fire only when particularcolumns are affected. The trigger will ignore changes that do not affect one of the listed columns.
The TRigger-action is further defined as
[FOR EACH ROW | FOR EACH STATEMENT] [WHEN expression]
BEGIN
trigger-step; [trigger-step;] *
END
At present only FOR EACH ROW TRiggers are supported, so each trigger stepwhich may be an INSERT,UPDATE, or DELETE statement or SELECT with a function expressionis performed once for every affected row inthe transaction that causes the trigger to fire. The WHEN clause can be used to cause a trigger to fire only for rowsfor which the WHEN clause is true. The WHEN clause is formed in the same way as the WHERE clause in aSELECT statement.
The WHEN clause and any trigger-steps may reference elements of the affected row, both before and after thetrigger action is carried out, as OLD.column-name and NEW.column-name respectively. For an UPDATE actionboth OLD and NEW are valid. An INSERT event can only provide a reference to the NEW value, whereas onlyOLD is valid for a DELETE event.
An ON CONFLICT clause can be specified in a trigger-step; however, any conflict resolution algorithm specified inthe statement that causes the trigger to fire will override it.
As you might expect, the syntax to drop a trigger is simply
DROP TRIGGER [database-name .] table-name
If you forget the name of a trigger, you can query sqlite_master using type = 'trigger' to find all the triggers on thecurrent database.
Using Triggers
In the last chapter we mentioned that triggers could be used to implement a cascading delete, so that rows from atable that referenced a foreign key would also be deleted if the foreign key were deleted from its own table. Thetrigger in the following example shows how this could be implemented on the demo database to delete entries fromthe timesheets table if the foreign key project_code is deleted from the projects table.
sqlite> CREATE TRIGGER projcode_cascade ...> AFTER DELETE ON projects ...> BEGIN ...> DELETE FROM timesheets WHERE project_code = OLD.code; ...> END;
Similarly, we could create a trigger that maintains data integrityif the project code changes in the projects table, thechild records in timesheets will be updated to reflect the new foreign key value.
sqlite> CREATE TRIGGER projcode_update ...> AFTER UPDATE OF code ON projects ...> BEGIN ...> UPDATE timesheets ...> SET project_code = NEW.code ...> WHERE project_code = OLD.code; ...> END;
A quick test verifies that this trigger is working as we want it to:
sqlite> UPDATE projects ...> SET code = 'NEWCODE' ...> WHERE code = 'ABCCONS';sqlite> SELECT count(*) ...> FROM timesheets ...> WHERE project_code = 'NEWCODE';count(*)
----------
3
Interrupting a Trigger
Within the trigger-steps it is possible to interrupt the command that caused the trigger to fire and execute one of theconflict resolution algorithms available in an ON CONFLICT clause. This is done using the RAISE() function, whichcan be invoked using a SELECT statement as one of the following:
RAISE (ABORT, error-message) |
RAISE (FAIL, error-message) |
RAISE (ROLLBACK, error-message) |
RAISE (IGNORE)
Issuing an ABORT, FAIL, or ROLLBACK within a trigger will cause the transaction to exit and take the relevantaction, and the error-message parameter is returned to the user.
We could use this behavior to prevent a project code from being deleted from the projects table while rows exist intimesheets that use it as a foreign key, rather than the rather destructive cascading delete.
sqlite> CREATE TRIGGER projcode_rollback ...> BEFORE DELETE ON projects ...> WHEN OLD.code IN ( ...> SELECT project_code FROM timesheets ...> ) ...> BEGIN ...> SELECT RAISE(ROLLBACK, 'Timesheets exist for that project code'); ...> END;
An attempted DELETE will produce an error:
sqlite> DELETE FROM projects WHERE code = 'NEWCODE';SQL error: Timesheets exist for that project code
We can also verify that the DELETE transaction was rolled back.
sqlite> SELECT * FROM projects ...> WHERE code = 'NEWCODE';code client_id title start_date due_date
Using RAISE(IGNORE) causes the current trigger to be abandoned; however, any changes made up to that pointwill be saved and if the trigger was fired as the result of another trigger, that outer trigger's execution will continue.
Creating a Trigger on a View
The syntax for using triggers on views is slightly different than with tables. This functionality is provided as a way ofintercepting INSERT, UPDATE, and DELETE operations on a view, usually to simulate that action by executing theactual steps necessary to make the requested data change appear in the view. Because a view may join two or moretables, a number of steps may be required.
The syntax for creating a trigger on a view is
CREATE [TEMP | TEMPORARY] TRIGGER trigger-name
INSTEAD OF database-event ON [database-name .] view-name
trigger-action
As before, database-event may be DELETE, INSERT, UPDATE, or UPDATE OF column-list, and thetrigger-action is one or more SQL operations contained between the keywords BEGIN and END.
Working with Dates and Times In our sample database we have chosen to use integers for columns that store a date value, represented by the formatYYYYMMDD. This format is fairly readable and, because the most significant part (the year) comes first, allowsarithmetic comparisons to be performed. For instance just as February 29th 2004 is earlier than March 1st,20040229 is a smaller number than 20040301.
This technique is not without its limitations. First, there is no validation on the values stored. Although February 29this a valid date in the leap year 2004, it does not exist three years out of four and the value 20050229 is not a realdate, yet could still be stored in the integer column or compared to a real date.
In fact even if you used a trigger to make the number eight digits long and also fall within a sensible year range, thereare many values that could still be stored that do not represent dates on the calendar. Very strict checking would berequired in your application program to ensure such date information was valid.
Similarly, you cannot perform date arithmetic using integer dates. Although 20040101 + 7 gives a date seven dayslater, 20040330 + 7 would give a number that looks like March 37th.
We have not even looked at a data type to store a time value yet, but the same limitations apply if a numeric field isused. SQLite contains a number of functions that allow you to work with both dates and times stored as characterstrings, allowing you to manipulate the values in useful ways.
Valid Timestring Formats
SQLite is fairly flexible about the format in which you can specify a date and/or time. The valid time string formats areshown in the following list:
YYYY-MM-DD
YYYY-MM-DD HH:MM
YYYY-MM-DD HH:MM:SS
YYYY-MM-DD HH:MM:SS.SSS
HH:MM
HH:MM:SS
HH:MM:SS.SSS
now
DDDD.DDDD
For the format strings that only specify a time, the date is assumed to be 2000-01-01. Where no time is specified,midday is used. Simply using the string now tells SQLite to use the current date and time.
The format string DDDD.DDDD represents a Julian day numberthe number of days since noon on November 24,4714 BC, Greenwich Mean Time. SQLite uses Julian date format internally to manipulate date and time values.
Displaying a Formatted Date and Time
The core date and time function in SQLite is strftime(), which has the following prototype:
This function is based upon the C function strftime() and the format parameter will accept most, although not all, of thesame conversion specifiers. The following example shows how a date can be reformatted to MM/DD/YY formatusing strftime().
The modifier keywords can be written as either singular or plural. In the last of the preceding examples, we used 1second and 1 year rather than 1 seconds and 1 years for readability. SQLite does not understand English grammar,so either is always acceptable.
In these examples we have used the same output format as the original timestring to return the date information in aformat that can be recognized by SQLite. You should only format the date differently when you want to display it inyour application in a particular way.
To save having to enter the same format strings repeatedly when working with dates, SQLite provides fourconvenience functions that call strftime() with predefined formats.
Use date() to return a date with the format string %Y-%m-%d and time() to return a time as %H:%S. The functiondatetime() returns the date and time using these two formats combined. Finally julianday() uses the %J format specifierto return the Julian day number.
The arguments to all four functions are the same as strftime() except that the format argument is omitted. Thefollowing example uses datetime() to produce a more concise SQL statement:
Other modifiers allow you to adjust a date or time to the nearest significant value. Specifying start of month, start ofyear, or start of day will decrease the value given in timestring to midnight on the first of the month or year, or on thatday respectively.
When executed on any day during 2004, the start of year modifier returns 2004-01-01, as shown in the followingexample:
sqlite> SELECT datetime('now', 'start of year');2004-01-01 00:00:00
Modifiers are applied to timestring in the order they appear in the statement, as shown in the following example. Notethat had the second statement been executed on the last day of the month, the result would have been differentthestart of the following month would have been returned.
sqlite> SELECT datetime('now', 'start of month', '+1 day');2004-07-02 00:00:00
sqlite> SELECT datetime('now', '+1 day', 'start of month');2004-07-01 00:00:00
Any number of modifiers can be combined, giving you considerable power when working with dates and times. Forinstance, the last day of the current month can be found using three modifiers in succession.
sqlite> SELECT date('now', '+1 month', 'start of month', '-1 day');2004-07-31
Handling Different Time Zones
The locale settings of your system will determine which time zone is used when displaying dates and times; however,the underlying system clock will use Coordinated Universal Time (UTC), also known as Greenwich Mean Time(GMT)Greenwich Mean Time (GMT). Your time zone setting will specify a number of hours to be added to orsubtracted from the UTC value to arrive at the correct local time.
For instance, to find the local time in New York you have to subtract five hours from UTC, or four hours duringdaylight savings time. Even in Greenwich, the local time is UTC + 1 hour during the summer months.
To convert between UTC and local time values when formatting a date, use the utc or localtime modifiers. Thefollowing examples were run on a system with the timezone set to Eastern Standard Time (UTC 5 hours).
SQL92 Features Not Supported We finish this chapter on SQLite's implementation of the SQL language by looking at features of the ANSI SQL92standard that are not currently supported by SQLite.
Although the CREATE TABLE syntax permits an optional CHECK clause to be present, the CHECKconstraint is not enforced.
The keywords FOREIGN KEY are allowable in a CREATE TABLE statement; however, this currently hasno effect.
Subqueries must return a static data set, and they may not refer to variables in the outer queryalso known ascorrelated subqueries.
All triggers are currently FOR EACH ROW, even if FOR EACH STATEMENT is specified.
Views are read-only, even when they select only from one table. However, an INSTEAD OF trigger can fireon an attempted INSERT, UPDATE, or DELETE to a view and deal with the transaction in the desiredmanner.
INSTEAD OF triggers are allowed only on views, not on tables.
Recursive triggerstriggers that trigger themselvesare not supported.
The ALTER TABLE statement is not present; instead a table must be dropped and re-created with the newschema.
Transactions cannot be nested.
count(DISTINCT column-name) cannot be used. However, this can be achieved by selecting a count() froma subselect of the desired table that uses the DISTINCT keyword.
All outer joins must be written as LEFT OUTER JOIN. RIGHT OUTER JOIN and FULL OUTER JOINare not recognized.
The GRANT and REVOKE commands are meaningless in SQLitethe only permissions applicable are thoseon the database file itself.
Chapter 4. Query Optimization Now that you are familiar with the SQL syntax implemented by SQLite, let's take a look at the way in which yourtables are created and how the way your queries are written can affect the performance of your database.
Keys and Indexes In the preceding chapter you met the CREATE INDEX statement, used to add an index to one or more columns of adatabase table.
Indexes are used to preserve a particular sort order on a column within the database itself, enabling queries in whichthe WHERE clause references that column directly to operate much faster. Rather than scanning a table's data fromtop to bottom and pulling out just the rows that match the given criteria, the index tells SQLite exactly where to lookin that table to find the matching rows.
What an Index Does
A database index is not an abstract concept; you certainly have come across one before. Let's consider the index inthis book, for instance, and think about when you would use it.
If you were trying to find references to the API call sqlite_exec(), for example, you might already expect to find thisin Chapter 6, "The C/C++ Interface"but bear in mind that a computer program cannot think for itself. So for amoment pretend you don't have the benefit of this knowledge.
To make sure you find every reference in the book, the systematic and thorough approach is to start from page one,line one and read each page a line at a time until the end of the book, noting each reference as you come across it.Clearly, this would be a time-consuming process.
Fortunately the production team behind this book has created a comprehensive index, with topics and terms usedthroughout the book listed alphabetically and page references given for each entry. You've seen an index before andknow how to use one, so you already know that it's much quicker to look up sqlite_exec() in the index and jump toeach page in turn to find the topic you are looking for than to scan every single printed word.
How Indexes Work in SQLite
A database index works much the same way as the index in a book. Suppose you have a table containing a list ofnames and telephone numbers, and the data has been built up over time as you come into contact with new people.It's very likely that some of your acquaintances will share a surname, but certainly your records were not added to thetable in alphabetical order.
Imagine you want to find the details for someone called Brown. The SQL SELECT statement might look like this:
SELECT * FROM contacts
WHERE last_name = 'Brown';
Without an index, the database has to look at each record in the database in turn and compare it to the string Brown.The more popular you are, the larger your contacts database will be and the longer this query will take to complete.This process is known as a full table scan.
However, having an index on the last_name column means that SQLite will know how to sort the records in thattable alphabetically by surname, and consequently it can pick out the matching values of last_name without evenlooking at the other rows in the table.
Such an index could be created on the contacts table as follows:
CREATE INDEX contacts_last_name ON contacts(last_name);
Remember that each index has to be given its own unique identifier, and that an index name cannot share the samename as a table, view, or trigger in the same database.
Note
Although the CREATE INDEX syntax allows you to specify a sort orderASC or DESCall indexes are currentlycreated in ascending order. This feature is reserved for possible future use.
Indexing Multiple Columns
An index can be created on more than one column at a timeknown as a clustered index. The order in which thecolumns are specified in the CREATE INDEX statement determines the sort order for the indexrecords are sorted onthe first column initially, then on the second column where values of the first column are the same.
Think of how a telephone directory is organized. People appear in order of their surname, but because there aremany people with the same surname, the listings are further ordered by first name or initials.
In SQLite, this kind of index for the contacts table would look like this:
CREATE INDEX contacts_name ON contacts(last_name, first_name);
A WHERE clause with conditions on both columns would find results very quickly using this index. In fact, becausethe primary sort order is last_name, a query that only references last_name would also perform well using this index.
The ordering of the columns for a clustered index is important. A condition imposed on first_name only would be nobetter off using this index than it would doing a full table scan.
Unique Indexes
You can specify the optional keyword UNIQUE in the CREATE INDEX statement to indicate that each value in theindexed columnor each permutation of values in a clustered indexonly appears in the table once.
Using CREATE UNIQUE INDEX is the same as imposing a UNIQUE constraint in the table schema, but it can bedone at any time after the table has been created. If you attempt to insert a duplicate value, SQLite will return an errorand the insert will fail.
In fact, using the UNIQUE attribute in the CREATE TABLE statement causes SQLite to create a unique indexautomatically on this column as shown in the following example. Note that the sql column is empty because noCREATE INDEX command was entered.
sqlite> create table mytable ( ...> a INTEGER PRIMARY KEY ...> );sqlite> select * from sqlite_master where type = 'index';type name tbl_name rootpage sql
If a column is declared as INTEGER PRIMARY KEY, it is used internally as the key for the table structure, so therecords in the table are ordered on that column by default. Although it does not cause an index to show up insqlite_master, the column will behave as if it were indexed. Explicitly creating an index on an INTEGER PRIMARYKEY column uses additional database resources without providing any extra benefits.
When to Create an Index
An index can be created or dropped at any time without affecting the data contained in that table. A number offactors determine when it becomes beneficial to create an index.
Usually you will create indexes on columns used in conditions in WHERE statements. Whether the purpose of theindex is to restrict a dataset from one table or to perform a join, generally speaking SQLite will be able to do its workmuch faster if key fields are indexed.
However, very small tables are often not made any faster by adding an index and in some cases, the index canactually slow things down. It can be just as fast for SQLite to do a quick scan of all rows than to look up the sortorder from the index.
If a table has data inserted frequently but not queried very often, use indexes sparingly. Although indexes can speedup queriesand UPDATE statements with a WHERE clause that references the indexed columninserting data is slowerwhen an index is present. In addition to adding a record to the table, an INSERT requires the index to be updated toreflect the position of the new data within the sort order of each indexed column.
Think of the filing cabinet in a traditional office, with customer files arranged alphabetically in the drawers. When anew document is filed, it takes a moment to locate the correct drawer and file and make sure that the item is put awayin the correct place. However, when it comes to accessing documents in the future, everything relating to a particularcustomer is in the same place and can be found very quickly.
The alternativea large stack of paperslets you file things away much faster by just placing them on top of the pile, butlooking for specific information becomes a daunting task. Just as most offices use a filing cabinet rather than anunsorted pile, most database tables can benefit from indexes when used correctly.
Don't go overboard with the number of indexes you create. The biggest performance benefits come from carefulindex selection based on the nature of the data stored in tables and the ways in which the application interacts with it.
If you do have too many columns indexed, it is possible that SQLite will not pick the best one for any particularquery. Only one index can be used on each table at a time, so a clustered index is the only way to apply thecapabilities of an index across two columns simultaneously. We will see an example of this later.
When Indexes Can Be Used
You also need to bear in mind what kinds of conditions can and cannot make use of an index. Most operators willuse an index if one is available provided that the column value is not modified before it is compared.
The following examples will all make use of an index on mytable.myfield if there is one.
SELECT * FROM mytable WHERE myfield = 'xyz';
SELECT * FROM mytable WHERE myfield > 2000;
SELECT * FROM mytable WHERE myfield IN ('val1', 'val2');
SELECT * FROM mytable WHERE myfield IN (SELECT somefield FROM anothertable)
However, the BETWEEN and LIKE operators will not use an index at any time. Such queries can usually berewritten if using an available index would be beneficial. For instance
SELECT * FROM mytable WHERE myfield BETWEEN 10 and 20;
becomes
SELECT * FROM mytable WHERE myfield >= 10 AND myfield <= 20;
Similarly, a LIKE operator that matches the first few characters of a string can be written using two inequalityoperators.
SELECT * FROM mytable WHERE myfield LIKE 'sql%';
becomes
SELECT * FROM mytable WHERE myfield >= 'sql' AND myfield < 'sqm';
Note
A LIKE operator with characters after a wildcard symbol cannot be rewritten as a pair of inequalities. This would belike trying to find people whose surnames end with a "y" in the telephone directory.
These examples show that two conditions on the same column combined with an AND can still make use of an index.However, this is not true of the OR keywordthe IN operator should be used instead wherever possible.
SELECT * FROM mytable WHERE myfield = 'abc' OR myfield = 'xyz';
becomes
SELECT * FROM mytable WHERE myfield IN ('abc', 'xyz');
If some operation is performed on a column value in the condition, SQLite will not be able to use an index. Forinstance, the following queries will be evaluated using a full table scan, regardless of any indexes on myfield:
SELECT * FROM mytable WHERE myfield % 2 = 1;
SELECT * FROM mytable WHERE substr(myfield, 0, 1) = 'w';
SELECT * FROM mytable WHERE length(myfield) < 5;
Tweaking the Table List
To join two or more tables together, the rows have to be combined before the results are returned, and the SQLiteengine does this using nested loops. The hierarchy of the loops is determined by the order in which table names arespecified in the FROM list, so that the leftmost table becomes the outer loop and the rightmost table becomes theinner loop.
Knowing a little about the tables you are performing the SELECT on can help you use this feature of SQLite toincrease the speed of the query. If you use the tables that will produce the smallest number of rowsafter any WHEREclause conditions that directly apply to that table have been appliedthe outer loop will be as small as possible, andconditions evaluated on the inner loops will be performed a smaller number of times.
Benchmarking
Listing 4.1 shows a simple benchmarking shell script that can be used to execute a query repeatedly within sqlite andusing the Unix time command to time execution.
Listing 4.1. benchmark.sh
#!/bin/sh
if [ $# -ne 3 ]; then
echo "Usage: $0 database-file sql-file num-loops"
exit;
fi
dbname=$1;
sqlfile=$2;
numloops=$3
tmpfile=/tmp/sqlite_benchmark.sql
> $tmpfile
echo "Generating temporary SQL file"
for i in 'seq 1 $numloops'; do
cat $sqlfile >> $tmpfile
done
echo "Executing query $numloops times"
time sqlite $dbname < $tmpfile > /dev/null
rm -f $tmpfile
This script is quite crude but will do enough to tell us how fast one query is in relation to another. A shell script hasbeen used here so as not to tie it to any particular programming API, but after you have read the chapters in Part II ofthis book, "Using SQLite Programming Interfaces," you will be able to create a benchmarking process that suits theenvironment you work in.
Note
The script in Listing 4.1 uses the seq command, which is part of the GNU coreutils package, included on Linux andcertain other Unix systems as standard. If your operating system does not include this command, the package can beobtained from http://ftp.gnu.org/pub/gnu/coreutils/.
The shell script version can be quite slow because it involves a lot of disk writes to generate the temporary SQL file.Using a programming API, the query could be executed repeatedly in a loop construct within the same SQLiteconnection.
However, to perform a loop in the shell with sqlite called multiple times to execute the same query over and overagain would carry a large overhead as the program has to start up, and open and close the database file for eachiteration. As we are only interested in the time taken to perform the query, the file containing multiple queries is builtup before sqlite is invoked and this work is not timed.
Create a text file containing the query to be analyzed and execute the program using the following syntax. Note thatbecause the SQL query is passed to sqlite many times from one file, the terminating semicolon must be present.
./benchmark.sh database-file sql-file num-loops
The output will look something like this:
$ ./benchmark.sh words somefile.sql 10000Generating temporary SQL file
Executing query 10000 times
real 0m6.814s
user 0m2.360s
sys 0m1.220s
Ideally the number of loops should be as high as you can bear to wait for the result to obtain a reliable averageexecution time. This will depend on the actual execution time of your query. The preceding example looped 10,000times and took just a few seconds, but for a slow query even one loop might take longer than this.
Some Examples
The examples that follow use some relatively large sample tables to demonstrate that indexes can be used effectively.You can download a sample of the database from the Sams Publishing website at http://www.samspublishing.com/.Enter this book's ISBN (without the hyphens) in the Search box and click Search. When the book's title is displayed,click the title to go to a page where you can download the code if you want to work through these examples.
The first example has three tables, each containing 10,000 rows. This is not large compared to the sizes of databasesSQLite is capable of handling, but is big enough for us to see the difference that an index can make.
The tables each contain a numeric integer between 1 and 10,000 and one word pulled randomly from a dictionaryfile. Table t1 uses an INTEGER PRIMARY KEY and contains each value from 1 to 10,000 in the id column. Theschema for t1 is as follows:
CREATE TABLE t1 (
num INTEGER PRIMARY KEY,
word TEXT NOT NULL
);
Though word is not defined as UNIQUE in t1, the sample table was created in such a way that values of word arenot duplicated. Therefore this table contains a list of 10,000 random English words, numbered sequentially.
CREATE TABLE t2 (
num INTEGER NOT NULL,
word TEXT NOT NULL
);
Table t2 contains one record for each value of word in t1; however, the num column contains a random integerbetween 1 and 10,000. As there is no UNIQUE constraint on num, the random way in which the data was generatedhas resulted in some numbers being duplicated and some being omitted from this table.
CREATE TABLE t3 (
num INTEGER NOT NULL,
word TEXT NOT NULL
);
The final table, t3, has a similar schema to t1 but with no INTEGER PRIMARY KEY. In fact data has been insertedso far in such a way that num is unique and every value from 1 to 10,000 is present, but to start with we do not wantthis column to be indexed.
Values of word were inserted randomly from the same dictionary file, which actually contains more than 15,000words. Therefore some words from t1 appear twice or more in t3, some are omitted, and some values that do notappear in either of the other tables have found their way into t3.
This data is trivial, but it allows us to create some reasonably large joins on both num and word to see how indexescould be used. In the queries that follow, we use count(*) as the selected output because the number of rows returnedwill usually be large and returning the count will not significantly affect the query's execution time.
Let's start by looking at how the INTEGER PRIMARY KEY in t1 makes things faster, even though there is novisible index. Because the records in t1 are stored in the order of the num column, a comparison on this column ismuch faster than on t2 where the ordering of num is random.
Listing 4.2 shows query t1.sql for performing a very simple benchmark on these tables using the equals operator.
Listing 4.2. t1.sql
SELECT count(*) FROM t1
WHERE num = 3500 AND 4000;
The result of executing this query 1000 times is
$ ./benchmark.sh words t1.sql 1000Generating temporary SQL file
Executing query 1000 times
real 0m0.007s
user 0m0.000s
sys 0m0.010s
File t2.sql performs the exact same query on table t2. Here are the results:
$ ./benchmark.sh words t2.sql 1000Generating temporary SQL file
Executing query 1000 times
real 0m11.261s
user 0m9.270s
sys 0m1.930s
As you can see, the difference the INTEGER PRIMARY KEY makes is vast because SQLite knows for certainwhat the order of the records will be.
When table t3 was created, records were actually inserted in order of column num, but without the column beingindexed, SQLite still has to perform a full table scan to be sure that it does not leave out any rows meeting thecondition.
$ ./benchmark.sh words t3.sql 1000Generating temporary SQL file
Executing query 1000 times
real 0m10.898s
user 0m8.880s
sys 0m1.970s
If we create an index on t3.num, the benchmark results show speeds comparable to the those of the original queryusing the INTEGER PRIMARY KEY:
sqlite> CREATE INDEX t3_num_idx ON t3(num);$ ./benchmark.sh words t3.sql 1000Generating temporary SQL file
Executing query 1000 times
real 0m0.029s
user 0m0.030s
sys 0m0.000s
Joining any two of these tables produces a potential 100 million rows, so this is where indexes can make a realimpact. Consider the query in Listing 4.3, which joins tables t1 and t2 on the num field. As t2.num is not indexed, afull table scan must be performed on t2 for each row in t1.
Listing 4.3. joint1t2.sql
SELECT count(*) FROM t1, t2
WHERE t1.num = t2.num;
This query is unacceptably slow, in fact so slow that it's not even worth running multiple times with the benchmarkscript. Just one execution took over a minute to complete.
$ time sqlite words < joint1t2.sql10000
real 1m3.822s
user 1m1.000s
sys 0m0.170s
If we join t1 and t3 instead using the same condition, this is much faster. As each row in t1 is iterated, SQLite can findthe corresponding record in t3 almost immediately using the index we just created.
$ ./benchmark.sh words joint1t3.sql 100Generating temporary SQL file
Executing query 100 times
real 0m4.721s
user 0m4.400s
sys 0m0.310s
Remember that we said the order in which tables appear in the FROM list is significanthere's an example. Let's take Listing 4.3 and reverse the table order as shown in Listing 4.4. This time, each row in t2 is examined in turn andjoined to t1 using the condition specified. Because t1.num is an INTEGER PRIMARY KEY, SQLite can jumpstraight to the row in question.
Listing 4.4. joint2t1.sql
SELECT count(*) FROM t2, t1
WHERE t1.num = t2.num;
$ ./benchmark.sh words joint2t1.sql 100Generating temporary SQL file
Executing query 100 times
real 0m4.435s
user 0m3.960s
sys 0m0.380s
Let's see what difference an index can make to the word column, which contains a variable-length character string,using the query shown in Listing 4.5.
Listing 4.5. t1word.sql
SELECT count(*) FROM t1
WHERE word = 'tickles';
$ ./benchmark.sh words t1word.sql 1000Generating temporary SQL file
Executing query 1000 times
real 0m8.533s
user 0m6.370s
sys 0m2.070s
If we add an index on t1.word and rerun the same test, the performance is much, much better.
sqlite> CREATE INDEX t1_word_idx ON t1(word);$ ./benchmark.sh words t1word.sql 1000Generating temporary SQL file
Executing query 1000 times
real 0m0.185s
user 0m0.160s
sys 0m0.030s
Our sample query just looks for a single word; in t1 and t2 the word "tickles" appears exactly once, so we could usea unique index if we want to. However, in t3 it actually appears three times, so trying to create a unique index gives awarning:
sqlite> CREATE UNIQUE INDEX t3_word_idx on t3(word);SQL error: indexed columns are not unique
Instead we have to use a nonunique index on t3.word.
The EXPLAIN Statement There is a way to compare how different indexes affect the speed of certain queries without having to performendless benchmark tests. However, it requires a little knowledge of the inner workings of SQLite.
The EXPLAIN command can be used to find out how an SQL query is parsed by SQLite and from that you candetermine the way in which it will actually be executed. The output from EXPLAIN is a series of opcodes from theVirtual Database Engine, which we'll look at in more detail in Chapter 10, "General Database Administration."
When using sqlite the .explain command can be used to make the format of the output of EXPLAIN more readable.The following example shows the opcodes used to process a query on t2 using a WHERE condition on thenon-indexed num column.
sqlite> .explainsqlite> EXPLAIN SELECT word FROM t2 WHERE num = 1234;addr opcode p1 p2 p3
To work out whether an index is used or not, knowing the actual meaning of all these opcodes is not necessary. If youcompare the preceding output to that for a similar query on t3 where num is indexed, you'll see the name of the indexin the p3 column alongside an OpenRead opcode.
sqlite> EXPLAIN SELECT word FROM t3 WHERE num = 1234;addr opcode p1 p2 p3
Because table and column names from the query will also appear in the p3 column, it is a good idea to have a strongnaming convention for your indexes so that they stand out in this output. In this case we used t3_num_idx, which tellsus exactly which table and column the index applies to.
We can use EXPLAIN to see which index SQLite chooses to use where two are availableremember that only oneindex can be used per table in a query. In the following examples, only the lines of output from EXPLAIN thatcorrespond to an index name are given.
sqlite> EXPLAIN SELECT * FROM t3 ...> WHERE num = 4000 AND word = 'soccer';...
6 OpenRead 1 1682 t3_word_idx
...
So of the two indexes available on table t3, SQLite has chosen the index on t3.word to use for this query.
Let's take a moment to recap what indexes are in place on these tables. You can always find this information byquerying the sqlite_master table:
sqlite> SELECT name, sql FROM sqlite_master ...> WHERE type = 'index';name sql
In fact the index that SQLite will choose where more than one is equally suitable is the one that appears last in thesqlite_master tableusually the one most recently created. A brand new index will also take precedence over olderindexes for the duration of the current SQLite session and the database schema has not been re-read. A clusteredindex is considered more suitable than an index on a single column if all the columns in the index form part of theWHERE clause.
There may be occasions when you want to prevent use of the index SQLite would otherwise choose in order toforce another index to be used. This is achieved by using an operator that does not affect the column's value.
To make sure the index on t3.word is used, add zero to the num column:
sqlite> EXPLAIN SELECT * FROM t3 ...> WHERE num+0 = 4000 AND word = 'soccer';...
6 OpenRead 1 1695 t3_word_idx
...
Similarly, to force use of the index on t3.num, use the concatenation operator to append an empty string to word:
sqlite> EXPLAIN SELECT * FROM t3 ...> WHERE num = 4000 AND word||'' = 'soccer';...
6 OpenRead 1 1077 t3_num_idx
...
If multiple indexes can be used in a query, each one will appear in the output from EXPLAIN. The following exampleshows that indexes are also used for a LEFT OUTER JOIN.
sqlite> EXPLAIN SELECT count(*) FROM t1 ...> LEFT OUTER JOIN t3 ON t1.num = t3.num ...> WHERE t1.word in ('apple', 'banana');11 OpenRead 2 927 t1_word_idx
15 OpenRead 3 2042 t3_num_idx
Using Transactions
You saw in Chapter 3, "SQLite Syntax and Use," that every change to the database must take place within atransaction, and that if one is not started explicitly with the BEGIN TRANSACTION command, SQLite will begin animplicit transaction whenever an INSERT, UPDATE, or DELETE statement is processed.
However, taking the time to create your own transactions can be well worth the effort. Each transaction requiresseparate low-level operations to open and close the journal file, and this is an unnecessary overhead when a numberof changes to the database can be made in a single transaction.
Therefore rather than letting SQLite start several implicit transactions for you when performing a series of changes tothe database, get into the habit of using BEGIN TRANSACTION and COMMIT TRANSACTION wherever suchstatements can be grouped together.
Note
If you are writing to a temporary table, the impact of using transactions is not so drastic. Because temporary tablesare not designed to be stored beyond the current session, the disk writes are not flushed to the database file asregularly.
The VACUUM Statement
Any kind of disk access is faster if the data being read is contiguous on the drive, and fetching rows from SQLite isno exception.
When records are deleted from a table, SQLite blanks out the space they used to occupy but does not remove itfrom the database file. The same applies when a table or index is dropped. Because additional disk space does notneed to be reallocated to the database file, a future INSERT statement that can occupy this space is slightly quicker,but not all data records are the same size, and before long the database will become fragmented.
The VACUUM command is SQLite's defragmenter. It works by systematically copying the contents of the databaseto a temporary fileleaving no empty spaceand then reloading the original file from this copy.
The SQL syntax allows a table name or index name argument to be passed to VACUUM for backwardcompatibility with much older versions of SQLite, however any argument given is now simply ignored. Issuing aVACUUM command will always clean up the entire database.
Tuning the Database Itself
This chapter has mostly covered optimizations that can be performed on individual SQL statements. Another way toimprove the performance of SQLite is to tune certain database parameters to suit your application.
We will examine system-wide performance considerations in Chapter 10.
Part II: Using SQLiteProgramming Interfaces
5 The PHP Interface
6 The C/C++ Interface
7 The Perl Interface
8 The Tcl Interface
9 The Python Interface
Chapter 5. The PHP Interface PHP is a server-side embedded scripting language for writing dynamic web pages that supports communication withmany different databases, and SQLite is no exception.
In this chapter we will look at how SQLite support is activated in PHP, how to communicate with a database througha web page, and how custom functions can be added to SQLite through the PHP interface.
Configuring PHP for SQLite Support Modern versions of PHP make it very easy to add in SQLite support, so we'll briefly look at the steps required forboth Linux/Unix and Windows versions of PHP.
For more general assistance with PHP configuration on a particular platform, refer to the online documentation at http://www.php.net/manual/en/installation.php.
Configuring PHP for Linux/Unix
From PHP 5, the SQLite extension is bundled with the PHP distribution and will be enabled at compile time unless itis disabled with the --without-sqlite option.
You do not need to install any SQLite components before building PHP, but you will need to download the sqlitetool separately if you want to use it to examine your databases outside of PHP.
For earlier PHP versions, the official extension can be obtained from http://pecl.php.net/package/SQLite. This alsoincludes source for libsqlite, so again there are no other prerequisites for adding the module to PHP. With PHP 4.3.0and later, you can use the pear utility to automatically download and install the SQLite extension:
The full output is quite lengthy, showing that the latest version of the extension has been downloaded, compiled, andcopied to your PHP extensions directory.
For earlier versions of PHP, you can download and compile the extension by hand after downloading the latestversion by following these steps:
# gzip d SQLite-1.0.2.tar.gz# tar xf SQLite-1.0.2.tgz# cd SQLite-1.0.2# phpize# ./configure# make# make install
With the extension installed, all you need to do to activate the new extension is add the following line to your php.inifile and restart the web server.
[sqlite]
extension="sqlite.so"
Alternatively you can also load the SQLite extension dynamically in each script that requires it with the dl() function.
dl("sqlite.so");
There should be no need to specify the full path to sqlite.so; the shared object file will be installed to the extension_dirvalue in php.ini.
Configuring PHP for Windows
The Windows DLL version of the SQLite extension is bundled with PHP5, and for earlier versions it can bedownloaded from http://snaps.php.net/win32/PECL_STABLE/php_sqlite.dll.
Save this file to the PHP extensions directoryusually C:\php\extensionsand then activate the module by adding thefollowing lines to your php.ini file and restarting Apache.
[sqlite]
extension=php_sqlite.dll
Checking for SQLite Support
If the SQLite extension is loaded, you can see some information about the module in the relevant section of the pagegenerated by the phpinfo() function. Figure 5.1 shows the typical output when SQLite support is present.
Figure 5.1. Output of phpinfo() showing SQLite support in PHP.
[View full size image]
In the command-line PHP, you can check which modules are loaded with the m switch, for instance:
$ php -m | grep sqlitesqlite
You can also check whether the extension is loaded programmatically using the extension_loaded() function, asdemonstrated in Listing 5.1. This listing will attempt to load the extension if it is not already present, and exit if theattempt fails.
Listing 5.1. Check to Find Whether SQLite Extension is Available to PHP
<?php
if (!extension_loaded("sqlite")) {
if (!dl("sqlite.so")) {
echo "The SQLite extension could not be loaded. Exiting.";
exit;
}
}
// Continue with SQLite-dependent code here
?>
Getting Information About the SQLite Extension
When you are sure the SQLite extension is available in your version of PHP, there are two functions available toreturn information about the version and encoding of the attached library.
These are the values we saw in Figure 5.1 in the output of phpinfo(). The functions allow you to use the respectivevalues programmatically if you want to perform conditional actions based on the library version, for instance.
A call to sqlite_libversion() requires no arguments and returns a text string containing the version number of the linkedlibrary. On the system that the screenshot in Figure 5.1 was taken on, this would return the string 2.8.11. The functionsqlite_libencoding() would return iso8559 on the same system.
By default PHP builds libsqlite with ISO-8859-1 encoding. In fact, this is not strictly ISO-8859-1it operatesaccording to your system local settings. The alternative, UTF-8, is not fully supported by PHP at the time of writingand may produce unexpected problems.
Using the PHP SQLite Extension Now we know that our PHP supports the SQLite interface, let's look at the set of commands that are available forcommunicating with a SQLite database.
Opening a Database
The PHP function to open a SQLite database is sqlite_open() and its prototype is given next:
The filename parameter specifies the database name, and if the file specified does not exist it will be created. Arelative or absolute path to the database file can be provided; otherwise, PHP looks for filename in the same directoryas the script being executed. To open an in-memory database, use :memory:.
Although PHP can execute scripts from the command line, this chapter assumes that you are mostly using PHP in aweb environment, where file permissions can be problematic. In order for PHP to open or to select from a SQLitedatabase, the user under which the web server is running must have read permissions on the database file.
To perform an UPDATE, INSERT, or DELETE, that web server userid must also not only have write permissionson the database file itself but also must be able to create files within the same directory so that the journal file can bewritten.
Apache often runs as the user nobody or apache and so care needs to be takenparticularly on a shared systemif thedatabase file is made writable to this user or, worse, if the file is made world-writable.
To prevent other users of the same web server from accessing your databases, PHP should be run in safe mode. Analternative is to use suPHPavailable from http://www.suphp.org/so that PHP can run as a number of different users onthe same server.
The mode parameter to sqlite_open() specifies the mode of the file, and is octal 0666 by default. The intended use ofthis parameter is to open a database file in read-only mode, using mode 0444 for example.
The resource returned is a database handle if sqlite_open() is successful. On failure, the function returns FALSE andif the optional errmessage parameter is passed by reference it will contain a descriptive error message explaining whythe database cannot be opened.
In Listing 5.2 we use sqlite_open() to open a new database webdb. To follow the rest of the examples in thischapter, make sure this database is created with the correct file permissions to be opened by PHP without any error.
Listing 5.2. Script to Open a Database with Error Checking
<?php
if (!$db = sqlite_open("webdb", 0666, &$errmessage)) {
echo "Could not open database:<br />\n";
echo $errmessage;
exit;
}
else {
echo "SQLite database opened";
}
?>
PHP allows for a persistent database connection to be used via the sqlite_popen() function, which takes the sameparameter list as sqlite_open().
Persistent connections reduce the overhead of repeatedly connecting to the same database. Where a persistenthandle has already been opened to the specified database, that handle is made available to the script. If no persistentconnection is available, a new connection is created.
Note
Persistent connections in PHP take advantage of the nature of multiprocess or multithreaded web servers. As long asPHP is built into Apache as a module or loaded into the web server using a supported SAPI, persistent connectionswill be available. If you are running PHP as a CGI wrapper, there is no persistence between page requests andalthough the sqlite_popen() function is still available, it will behave no differently than sqlite_open(). For moreinformation, see http://php.net/manual/en/features.persistent-connections.php.
To close a SQLite database session, use sqlite_close(). The only parameter is the database resource returned by thefunction that opened the connection. If sqlite_popen() was used, the persistent connection will be closed and will nolonger be available to other processes.
Passing Queries and Commands to SQLite
To pass an SQL command to SQLite, use the sqlite_query() function. Two parameters are requireda databaseconnection resource and the query text itself. The prototype is given here:
In fact the parameters can be specified in reverse order to that shown in the preceding example for compatibility withother PHP database extensions. However, the preferred syntax is to give the dbhandle parameter first for consistencywith other SQLite APIs.
Note
The terminating semicolon is not required when a single query is executed through sqlite_query(). However, the PHPextension does allow you to submit multiple commands at the same time by passing a single string of SQL statementswith each terminated by a semicolon.
For a SELECT statement, resource will be a result handle that can be processed by further functions, as we will seeshortly. The result will be FALSE if a SELECT query fails.
Where query does not return any rows, for instance an UPDATE, INSERT, or DELETE statement or a CREATETABLE as shown in the example in Listing 5.3, the resource value returned is trUE if the statement is executedsuccessfully; otherwise, it is FALSE.
To find the reason a query has failed, use the sqlite_last_error() and sqlite_error_string() functions to retrieve theerror message from SQLite. The former returns a numerical error code, which the latter decodes into ahuman-readable error message, as demonstrated in Listing 5.3.
Listing 5.3. Executing a CREATE TABLE Statement Through the PHP Extension
<?php
if (!$db = sqlite_open("webdb", 0666, &$errmessage)) {
echo "Could not open database:<br />\n";
echo $errmessage;
exit;
}
else {
$sql = "CREATE TABLE contacts (
id INTEGER PRIMARY KEY,
first_name CHAR,
last_name CHAR,
email CHAR
)";
$res = sqlite_query($db, $sql);
if (!$res) {
echo "Error with query:";
echo "<PRE>$sql</PRE>";
echo sqlite_error_string(sqlite_last_error($db));
}
else {
echo "Command successful";
}
}
?>
The first time Listing 5.3 is executed, the table contacts will be created. However, a second execution will produce anerror message similar to that shown in Figure 5.2.
Figure 5.2. Error messages displayed when a SQLite query fails.
[View full size image]
The screenshot was taken on a system that has display_errors turned on in php.ini, and you can see that the feedbackfrom the PHP extension is generally more useful than the SQLite error message returned.
Using Commands That Change the Database
The apostrophe or single quote character needs to be delimited before it can be stored in a SQLite column. Twoapostrophes appearing together are treated as a single character within a string enclosed by single quotes.
For instance this statement is invalid:
INSERT INTO mytable (myfield) VALUES ('It's tricky');
It should be rewritten as follows, which will insert the string It's tricky containing a single apostrophe into the database:
INSERT INTO mytable (myfield) VALUES ('It''s tricky');
The function sqlite_escape_string() can be used to automatically make strings compatible with SQLite, by escapingsingle quote characters and encoding binary-unsafe characters. Listing 5.4 contains a PHP script that loops through anarray of contact details and safely inserts each record into the database.
Note
The PHP function addslashes() is not suitable for escaping quotes in values intended for storage in SQLite.
Listing 5.4. Escaping Special Characters Before Inserting Data
In lines 3 to 11 we declare an array of the contact information that will be inserted. The outer loop, opened on line 20,extracts each of the inner arrays into $c, and then each value within $c is escaped before the query string is built.
As the contacts table includes an INTEGER PRIMARY KEY, a unique number is assigned to the id column eachtime an INSERT is performed. The value assigned can be retrieved using the sqlite_last_insert_rowid() function.
Replace line 36 in Listing 5.4 with the following code to display the value of the id column for each record inserted:
$id = sqlite_last_insert_rowid($db);
echo "Inserted $c[first_name] $c[last_name] as ID $id<br />\n";
Whenever an UPDATE, INSERT, or DELETE statement is issued, it's likely that one or more rows in a table havebeen modified in some way. The function sqlite_changes() will return the number of affected rows and can be used toconfirm that the command has done what it was intended to do.
Listing 5.5 issues an UPDATE statement converting each first_name and last_name value to uppercase in thecontacts table and shows the number of rows affected by this command.
Listing 5.5. Finding the Number of Rows Affected by an UPDATE Command
<?php
if (!$db = sqlite_open("webdb", 0666, &$errmessage)) {
echo "Could not open database:<br />\n";
echo $errmessage;
exit;
}
else {
$sql = "UPDATE contacts
SET first_name = upper(first_name),
last_name = upper(last_name)";
$res = sqlite_query($db, $sql);
if (!$res) {
echo "Error with query.";
echo "<PRE>$sql</PRE>";
echo sqlite_error_string(sqlite_last_error($db));
}
$changes = sqlite_changes($db);
echo "Updated $changes row(s)<br />\n";
}
?>
Note
The number reported by sqlite_changes() is the number of matching rows in the WHERE clause of an UPDATEstatement, or the total number of rows in the table if there is no WHERE. It is not necessarily the number of rowschanged by the command; where a row is updated to the same value it previously held, this is still counted in the totalnumber of changes.
Working with Retrieved Data
The resource returned by sqlite_query() for a SELECT statement can be used in conjunction with a number of PHPfunctions to fetch the data returned by the query and to find information about the dataset itself.
The functions sqlite_num_rows() and sqlite_num_fields() both take a single resource parameter and return an integervalue of the number of rows and columns in the returned dataset respectively.
To fetch a column name, use sqlite_field_name(). The function takes two parameters, the resource returned bysqlite_query() and a column identifier specified as an index number. Where the columns are selected in the FROM listof the query, the leftmost column in the list has index 0. If SELECT * FROM tablename is used, the columns arenumbered from zero in the order that they appear in the database schema.
The function to fetch a column value is sqlite_column(), which takes the same two parameters as sqlite_field_name(),but this time the column identifier may be either an index number or the name of the column, if it is known.
For instance, if the query passed was
SELECT first_name, last_name
FROM contacts;
the following two statements are equivalent:
$value = sqlite_column($res, 1);
$value = sqlite_column($res, "last_name");
Immediately after sqlite_query() has been executed, the values returned by sqlite_column() are from the very first rowof the dataset. To shift the result handle pointer to the next row, use sqlite_next().
The most common usage is to seek to each row in turn using sqlite_next() in a loop structure. To find out when theend of the dataset is approaching, use sqlite_has_more() with a single resource parameter, which returns trUE as longas there are still more rows to fetch. If you call sqlite_next() when there are no more rows to be fetched, a warningwill be shown.
The example in Listing 5.6 puts together the functions we have briefly introduced in this section to execute a queryand display the contents of the returned dataset in a formatted HTML table. The table column headings are thedatabase column names, and each record is displayed in a table row in turn until there are no records left.
Listing 5.6. Displaying the Results of a Query in an HTML Table Using PHP
<?php
if (!$db = sqlite_open("webdb", 0666, &$errmessage)) {
The output should look similar to that shown in Figure 5.3.
Figure 5.3. Output from Listing 5.6, showing the result of a query displayed in an HTML table.
[View full size image]
Let's look at parts of this code in more detail. After we have passed the query to SQLite using sqlite_query(), wedisplay a count of the number of rows that will be displayed in line 20.
Next, after opening an HTML <TABLE> tag, we generate a row of table headings<TH>by counting through thenumber of fields in the dataset and outputting the name of each one in turn.
We then use the same for loop as in the preceding lines within another while loop to fetch a row at a time and displayeach field value in turn inside <TD> tags, to align with the column headings generated previously.
In fact the outer while loop could also be performed as a for loop by counting up to the value returned bysqlite_num_rows(). The method you use will be a matter of preference.
Another way to fetch rows of data is using sqlite_current() or sqlite_fetch_array(). Both functions return anassociative array of all the values from the next row using the column names as keys, with the latter also advancing theresult handle pointer to the next row, as if sqlite_next() had been called.
After executing this command
$row = sqlite_current($res);
the following two statements are equivalent:
$value = sqlite_column($res, "first_name");
$value = $row["first_name"];
The default behavior of sqlite_fetch_array() and sqlite_current() is to return an associative array with columns indexedboth by their field name and their numerical position beginning at zero. Therefore if first_name was the first columnlisted, the following statement is also equivalent:
$value = $row[1];
An optional second parameter can be passed to either function to define this behavior. The valid constants areSQLITE_ASSOC, SQLITE_NUM, and SQLITE_BOTH, which instruct PHP to create an associative array, anumerically indexed array or an array with both sets of indexes respectively.
For instance, if the command executed is
$row = sqlite_current($res, SQLITE_NUM);
$row["first_name"] will not contain any value.
Note
The configuration directive sqlite.assoc_case in php.ini determines the case of the keys used in the associative arraysreturned by sqlite_current() and sqlite_fetch_array(). The default value is zero for mixed-case keys that match thecolumn names exactly. When this setting is 1 or 2, the key names are converted to all uppercase or all lowercaserespectively.
The following section of code could replace lines 30 to 37 of Listing 5.6, performing the same loop using do ... while:
while ($row = sqlite_fetch_array($res)) {
echo "<TR>\n";
for ($i=0; $i<sqlite_num_fields($res); $i++) {
echo "<TD>" . $row[$i] . "</TD>\n";
}
echo "</TR>\n";
}
If you are interested only in the first row returned by a query, the function sqlite_fetch_single() returns an associativearray of values from the very first row in the dataset. Because the result handle pointer is not moved and its positiondoes not affect the behavior of this function, sqlite_fetch_single() is the most optimal way to return data to PHP whenonly the first row is required.
The PHP extension also provides a single function for returning the entire result of a query into a single array. Thefollowing code line gives the prototype for sqlite_array_query(), and as with sqlite_query(), the order of the query anddbhandle parameters can be reversed for compatibility with other PHP database extensions.
array sqlite_array_query ( resource dbhandle, string query [, int result_type])
This function returns a single, two-dimensional array made up of one array of associative arrays of values for eachrecord in the dataset.
The same functionality could be implemented quite easily in your PHP script using a sqlite_query() and a loop onsqlite_fetch_array(), but as a built-in function of the PHP SQLite extension, it is significantly faster to use this functionthat builds the array in PHP. The following two pieces of code have the same result:
$data = sqlite_query($db, $sql);
$res = sqlite_query($db, $sql);
while($row = sqlite_fetch_array($res)) {
$data[] = $row;
}
The contents of the $data array that was generatedby either methodfor the query SELECT * FROM contacts isshown in Listing 5.7, using the print_r() function to recursively output the array elements.
Listing 5.7. Array Returned by sqlite_array_query() Function
The PHP extension allows random access of the rows in a dataset using the sqlite_seek() function, for which theprototype is given here:
bool sqlite_seek (resource result, int rownum)
The function attempts to set the result handle pointer to row rownum for the given resource, and returns trUE onsuccess or FALSE if the rownum specified does not exist.
Listing 5.8 shows how sqlite_seek() can be used to jump to a random row within the dataset returned from oursample contacts database. Execute this script a few times to see that a random selection is made each time.
Listing 5.8. Using sqlite_seek() to Jump to a Specific Row
<?php
if (!$db = sqlite_open("webdb", 0666, &$errmessage)) {
The function sqlite_rewind() takes a single resource parameter and resets the result handle pointer to the first record inthe dataset. The following two commands are equivalent:
sqlite_rewind($res);
sqlite_seek($res, 0);
For these operations to work, the result handle must be buffered and seekable, and creating this data structure inmemory is an overhead that can be avoided. The function sqlite_unbuffered_query() works just the same assqlite_query() except that the result handle can only be accessed sequentially. As only one row is returned at a time,this gives much better performance and is recommended unless random access is a necessity.
Working with User-Defined Functions A very powerful feature of the PHP SQLite extension is the ability to define functions in PHP that can be registeredand executed within an SQL statement. It is therefore possible for PHP developers to extend the functionality ofSQLite using a familiar language.
The example in Listing 5.9 defines a trivial function in PHP that reverses each word in a string. This differs from theusual behavior of strrev(), which reverses the entire string. The example uses sqlite_create_function() to register auser-defined function, word_reverse(), for use within SQLite.
Listing 5.9. Adding a User-Defined Function to SQLite in PHP
<?php
function reverse_words($string) {
$w = explode(" ", $string);
for($i=0; $i<count($w); $i++) {
if ($i > 0)
$ret .= " ";
$ret .= strrev($w[$i]);
}
return($ret);
}
if (!$db = sqlite_open("webdb", 0666, &$errmessage)) {
$sql = "SELECT word_reverse('The quick brown fox');";
$res = sqlite_query($db, $sql);
echo sqlite_column($res, 0);
}
?>P
First, we declare the function reverse_words() in PHP. Using explode() to break up the passed-in stringassumingwords are separated by a space characterwe then call strrev() on each word in turn and piece the string backtogether:
function reverse_words($string) {
$w = explode(" ", $string);
for($i=0; $i<count($w); $i++) {
if ($i > 0)
$ret .= " ";
$ret .= strrev($w[$i]);
}
return($ret);
}
After opening a SQLite connection to webdb as database resource $db, we use sqlite_create_function() to registerthe user-defined function.
There are four arguments to sqlite_create_function(). First the database resource, in this case $db, is supplied. Thesecond parameter is the function name that will be used by SQLite, and the third is the name of the PHP function fromwhich it is derived.
In our example, we used word_reverse() for the SQLite function and reverse_words() for the PHP function to tellthe two apart. There is actually no reason that they cannot share the same name if desired.
The fourth parameter is optional and indicates the number of parameters that are passed to the UDF. In this case,only a single string parameter is required, so the value is 1. This acts as a hint to the SQLite parser, and an error willbe returned if the number of parameters passed when the function is called does not match this value. If the PHPfunction can accept a variable number of parameters, this value can be omitted.
The output from Listing 5.9 is as you might expect:
ehT kciuq nworb xof
A function registered with sqlite_create_function() does not exist outside of the process in which it is executed.Because the function is created within a given database resource, a user-defined function can be called within PHPfunctions that have access to the same resource, so the scope of a UDF within a script is not an issue. However, thefunction is not saved to the database permanently.
Where a persistent database connection has been used, sqlite_create_function() must still be called again before thefunction is available to SQLite. If PHP reattaches to a persistent SQLite connection in which a user-defined functionhas been declared and you attempt to call that function, the following warning will be displayed rather than a no suchfunction error.
sqlite_query(): this function has not been correctly defined for this request
It is possible to use an existing function name as the name of a UDF in the second parameter tosqlite_create_function(), which will overload the behavior of that function without warning. All subsequent calls to thatfunction will execute the PHP function specified rather than the built-in function.
The php() Function
When PHP first opens a SQLite database, it automatically registers the function php(), which provides a quick wayto access any PHP function without having to add it as a UDF with sqlite_create_function(). Both built-in PHPfunctions and those defined in your script code can be referenced this way.
Listing 5.10 shows how the PHP function ucwords() can be called from within SQLite using php() to capitalize thefirst character of each word in a string.
Listing 5.10. Calling a PHP Function from SQLite Using php()
<?php
if (!$db = sqlite_open("webdb", 0666, &$errmessage)) {
echo "Could not open database:<br />\n";
echo $errmessage;
exit;
}
else {
$sql = "SELECT php('ucwords', 'The quick brown fox');";
$res = sqlite_query($db, $sql);
echo sqlite_column($res, 0);
}
?>
The output from Listing 5.10 is as expected:
The Quick Brown Fox
Creating Aggregating Functions
The process to create an aggregating user-defined functionone that applies a calculation across all the rows returnedby a query, or across similar rows indicated by a GROUP BY clauseis slightly different than a regular function.
The function sqlite_create_aggregate() requires two functions to manage the aggregate; one that is called for eachrow on the result set returnedthe step functionand another that is called once after all the rows have beenprocessedthe finalize function.
A context variable is passed by reference so that both functions can access the data being aggregated.
The example in Listing 5.11 creates and demonstrates a new aggregating function to calculate the median average ofa set of numbers. The median is a value such that half the values in the set are above the median and half are below it.By contrast, the built in avg() function computes the mean averagethe sum of all the values divided by the number ofitems in the dataset.
Listing 5.11. Adding an Aggregating UDF with sqlite_create_aggregate()
$sql = "SELECT median(num), avg(num) FROM numbers";
$res = sqlite_query($db, $sql);
echo "Median is ";
echo sqlite_column($res, 0);
echo "<br />\n";
echo "Mean is ";
echo sqlite_column($res, 1);
echo "<br />\n";
}
?>
Before we can use our new function we need to create the table numbers and insert some values. The script in Listing5.12 creates the table and inserts the squares of numbers one to nine in turn. Because of the geometric nature of thissequence, we can use it to show how the median and mean averages differ using our new function.
Listing 5.12. Inserting a Sequence of Square Numbers
<?php
if (!$db = sqlite_open("webdb", 0666, &$errmessage)) {
echo "Could not open database:<br />\n";
echo $errmessage;
exit;
}
else {
$sql = "CREATE TABLE numbers (num INTEGER)";
$res = sqlite_query($db, $sql);
for ($i=1; $i<=9; $i++) {
$val = $i * $i;
$sql = "INSERT INTO numbers (num) VALUES ($val)";
$res = sqlite_query($db, $sql);
}
}
?>
The following line of code shows the complete sequence of numbers inserted. From looking at this sequence innumerical order, we can see that the median will be 25, as there are four numbers on either side:
1 4 9 16 25 36 49 64 81
Let's look back through Listing 5.11 in more detail. The step function simply builds an array in PHP, adding eachvalue in turn:
function median_step(&$context, $num) {
$context[] = $num;
}
Notice however that $context is passed by reference in the function parameters. The value does not need to bereturned.
The finalize function has to do a little more work. After the array has been created, we sort it into numerical orderand find the number of items. Again, $context is passed by reference so that the array built by median_step() isavailable within this scope:
function median_finalize(&$context) {
sort($context);
$count = count($context);
If there are an odd number of values, the median is the middle value itself. If there are an even number of values, themedian is the mid-point between the two middle values.
To compare the median of the values in numbers with its average, this SQL statement is executed:
$sql = "SELECT median(num), avg(num) FROM numbers";
$res = sqlite_query($db, $sql);
The values displayed are
Median is 25
Mean is 31.6666666666667
Working with Binary Data in UDFs
If the data you are passing to or from a UDF may not be binary safe, you must encode it to avoid possible problems.A NUL byte is a string terminator, so if one could appear anywhere other than the end of a string you may getunexpected results, for instance.
The function sqlite_udf_encode_binary() takes a string parameter and returns a binary-safe encoded version. Thereverse is sqlite_udf_decode_binary(). For performance reasons, PHP will not perform such encoding automaticallyitself.
Using the PEAR Database Class In addition to the API we have already discussed, PHP also allows access to SQLite databases via the PEARDatabase class. We will show briefly how to use this with SQLite.
The PEAR class provides a unified API for accessing SQL-based databases. It supports SQLite as well as manyother popular databases such as Microsoft SQL Server, MySQL, Oracle, and PostgreSQL, as well as ODBCconnectivity.
The database-abstraction approach allows you to create scripts that can work on a wide range of database enginessimply by changing the connection parameters. The same set of functions is made to work with whatever databasetype you happen to be connected to at the timeright down to a function that delimits quotes and other specialcharacters the correct way for that database. If you expect your PHP application to need porting to a differentdatabase system in future, the PEAR class is well worth considering.
Note
Database-specific features must be worked around when using a database abstraction layer. For instance theINTEGER PRIMARY KEY feature specific to SQLite is similar to an AUTO INCREMENT field in MySQL, butthe same functionality can be only achieved in Oracle using CREATE SEQUENCE. In fact the PEAR Database classforces you to use a database-level sequence object for all autoincrementing fields for maximum compatibility.
To download and install the PEAR DB class automatically if you do not already have it, use the pear utility:
# pear install DB
There are two ways to define a database connection. The first uses a string to indicate the Data Source Name (DSN)and usually looks like this:
dbtype://username:password@hostname/dbname
For SQLite, however, the username, password, and hostname fields are not required, so the following DSNs aresufficient to identify a database file in the current directory or in the given path respectively.
sqlite:///dbfile
sqlite:////home/chris/sqlite/dbfile
The second way to define a DSN is by setting elements of an array of parameters, which is defined by the class asfollows:
$dsn = array(
'phptype' => false,
'dbsyntax' => false,
'username' => false,
'password' => false,
'protocol' => false,
'hostspec' => false,
'port' => false,
'socket' => false,
'database' => false,
);
Only the phptype and database elements are required to identify a SQLite database, so the following array is a validDSN:
$dsn = array(
'phptype' => 'sqlite',
'database' => 'webdb'
);
Connecting to the database with a DSN defined by either method is as simple as the following statement:
$db = DB::connect($dsn);
To execute an SQL command that does not return any rows, use the query() method:
$sql = "DELETE FROM mytable WHERE myfield = 'somevalue'";
$res = $db->query($sql);
To test if a query was successful, use the isError() function on the returned result:
if (DB::isError($res)) {
echo "An error occurred";
}
For a SELECT query that returns a single row, use the getrow() method to return that row as an arraysimilar to callingsqlite_fetch_single(). If the optional second parameter is passed the constant DB_FETCHMODE_ASSOC, anassociative array using the column names as indexes will be created:
$sql = "SELECT * FROM mytable";
$row = $db->getRow($sql, DB_FETCHMODE_ASSOC);
To return all the rows in a query as an arraysimilar to sqlite_array_query()use the getAll() method.
$sql = "SELECT * FROM mytable";
$row = $db->getAll($sql, DB_FETCHMODE_ASSOC);
This chapter has only touched on a few of the many functions available in the PEAR Database class. There iscomprehensive online documentation for the class at http://pear.php.net/manual/en/package.database.php.
Chapter 6. The C/C++ Interface In this chapter we will look at how SQLite's C/C++ interface can be incorporated into your programs. We willdiscuss the different ways of querying a database and look at how you can add your own custom functions to SQLite.
Preparing to Use the C/C++ Interface In order to develop software with an embedded SQLite database, you need to make sure your system has theSQLite development library installed. The latest version can always be downloaded from http://www.sqlite.org/download.html.
If you have already installed and used sqlite as a precompiled binary, you will not necessarily have the library yet. ForLinux, the precompiled library is sqlite.so.gz and for Windows it is sqlitedll.zip. If you also want to use the TCLbindings, precompiled libraries are available that support this, named tclsqlite.so.gz and tclsqlite.zip respectively.
Users of Red Hat Linux and other Unix-like systems that use the RPM package format can use the sqlite-develpackage to install the library along with header files and documentation.
If you compiled SQLite from source, the libraries were installed when you issued the make install command. Adefault source installation on Linux/Unix will put libsqlite in the /usr/local/lib directory and sqlite.h in /usr/local/include.
Using the C Language Interface Now that you have made sure the library is available on your system, let's take a look at the functions that make upthe C language interface.
Opening and Closing a Database
The function to open a SQLite database is sqlite_open(). The prototype is
sqlite *sqlite_open(const char *dbname, int mode, char **errmsg);
The dbname parameter is the name of the database file on the filesystem. If just a filename is given, it is assumed thatthe database is in the current directory at runtime. A relative or absolute path can be specified.
The mode parameter is intended for future use, to specify the file mode in which the database file is opened. Typicalvalues would be 0777 for read/write access or 0444 for read-only. However, at the time of writing, this parameter isignored by the library.
The sqlite structure to which a pointer is returned is an opaque structure that must be passed as the first parameter toall the other SQLite API functions. If the database cannot be opened, the return value will be NULL and errmsg willpoint to the location of the error message created by sqlite_open(). The memory allocated for errmsg should be freedusing sqlite_freemem().
To close the database, use sqlite_close(), which is simply passed the open sqlite structure pointer.
The example in Listing 6.1 shows how to establish a connection to a database called progdb in the current directory,and displays an error message if the database cannot be opened.
Listing 6.1. Connecting to a SQLite Database Using C/C++
fprintf(stderr, "Could not open database: %s\n", errmsg);
sqlite_freemem(errmsg);
exit(1);
}
printf("Successfully connected to database\n");
sqlite_close(db);
}
We compile listing6.1.c into an executable listing6.1 as follows:
$ gcc -o listing6.1 listing6.1.c lsqlite
Executing the new program should tell us that we have been able to connect to the progdb database. The first timeyou run the program, the database will be created and you will be able to see a file called progdb in the currentdirectory:
$ ./listing6.1Successfully connected to database
If there was a problem opening the database, for example, if you do not have write permission on the currentdirectory, an error will be displayed:
$ cd /$ /path/to/listing6.1Could not open database: unable to open database: progdb
Getting Information About the SQLite Library
Two constants are defined in sqlite.h that allow you to find information about the version and encoding of the library:
const char sqlite_version[]
const char sqlite_encoding[]
The value of sqlite_version is a string representation of the library version number, for instance 2.8.13.
The encoding method for the library is determined at compile time and the default value of sqlite_encoding is iso8859for ISO-8859-1 encoding. The alternative encoding method currently provided by SQLite is UTF-8. The encodingmethod used by a program cannot be changed without recompiling the library.
Executing SQL Statements
The function sqlite_exec() is used to send an SQL statement to the SQLite engine. Its prototype is
int sqlite_exec(
sqlite *db,
char *sql,
int (*xCallback)(void*,int,char**,char**),
void pArg,
char **errmsg
);
The five parameters required are as follows:
db is a valid sqlite structure pointer.
sql is a null-terminated character string containing one or more SQL statements.
The third parameter is a pointer to a callback function or may be NULL if no callback is required.
pArg is a pointer to the first argument of the callback function.
If the query fails, the error message from SQLite will be pointed to by errmsg.
The callback function is invoked once for each row returned by the query, and we will examine this shortly. An SQLstatement that is not a SELECT does not return any rows, so no callback function is required and the third and fourthparameters can be NULL.
Note
A semicolon is not required to terminate a single SQL query in the sql parameter. However, if multiple statements arepassed in one sqlite_exec() call, they must be separated with semicolons.
The example in Listing 6.2 shows how a CREATE TABLE statement is executed using the C interface.
Listing 6.2. Executing a CREATE TABLE Statement with the C Library
fprintf(stderr, "Could not open database: %s\n", errmsg);
sqlite_freemem(errmsg);
exit(1);
}
ret = sqlite_exec(db,
"CREATE TABLE contacts ( \n"
" id INTEGER PRIMARY KEY, \n"
" first_name CHAR, \n"
" last_name CHAR, \n"
" email CHAR)", NULL, NULL, &errmsg);
if (ret != SQLITE_OK)
{
fprintf(stderr, "SQL error: %s\n", errmsg);
}
sqlite_close(db);
}
The SQL statement in Listing 6.2 is formatted using \n characters and indented for readability. This is not arequirement, but as the schema stored in the database is the exact CREATE TABLE statement executed, theformatting here will make the output of the schema more readablefor instance when using the .schema command insqlite.
Running the compiled program should be silent, assuming there are no problems accessing progdb, and the table willbe created. If you execute the program a second time the table name will already be in use, and the following errormessage from SQLite will be displayed.
The constant SQLITE_OK is a zero value and is used to check for successful execution of an SQL statement. Anynon-zero value indicates that an error has occurred. The full list of return code constants is shown in Table 6.1.
Table 6.1. Return Code Constants
Constant Value Description
SQLITE_OK 0 Successful result.
SQLITE_ERROR 1 SQL error or missing database.
SQLITE_INTERNAL 2 An internal logic error in SQLite.
SQLITE_PERM 3 Access permission denied.
SQLITE_ABORT 4 Callback routine requested an abort.
SQLITE_BUSY 5 The database file is locked.
SQLITE_LOCKED 6 A table in the database is locked.
SQLITE_NOMEM 7 A malloc() failed.
SQLITE_READONLY 8 Attempt to write to a read-onlydatabase.
SQLITE_CORRUPT 11 The database disk image ismalformed.
SQLITE_NOTFOUND 12 Table or record not found.
SQLITE_FULL 13 Insertion failed because database isfull.
SQLITE_CANTOPEN 14 Unable to open the database file.
SQLITE_PROTOCOL 15 Database lock protocol error.
SQLITE_EMPTY 16 Database table is empty.
SQLITE_SCHEMA 17 The database schema changed.
SQLITE_TOOBIG 18 Too much data for one row of atable.
SQLITE_CONSTRAINT 19 Abort due to constraint violation.
SQLITE_MISMATCH 20 Data type mismatch.
SQLITE_MISUSE 21 Library used incorrectly.
SQLITE_NOLFS 22 Uses OS features not supported onhost.
SQLITE_AUTH 23 Authorization denied.
SQLITE_ROW 100 sqlite_step() has another row ready.
SQLITE_DONE 101 sqlite_step() has finished executing.
More information on each of these return codes can be found in Appendix E, "C/C++ Interface Reference."
Note
If you want to validate a string to see if it forms a valid SQL statement, use the sqlite_complete() function. Taking asingle character pointer argument, the function returns true when the string supplied forms a complete SQL statementor false if more text is required.
The sqlite_complete() function considers a statement to be complete if it ends with a semicolon, ignoring semicolonsthat separate statements within a CREATE TRIGGER statement. The function does not attempt to validate the syntaxof the SQL statement itself.
Using Commands That Change the Database
Now that we have created a table, let's insert some rows. Listing 6.3 creates a program that reads three values fromstdin that are used as the first_name, last_name, and email fields in a subsequent INSERT statement.
As the contacts table includes an INTEGER PRIMARY KEY field, a unique number is assigned to the id columneach time an INSERT is performed. The value assigned can be retrieved using the sqlite_last_insert_rowid() function.This statement is used to confirm that the insert took place successfully and shows the assigned id value.
Compile and run Listing 6.3 and enter some details:
$ ./listing6.3Enter first name: ChrisEnter last name: NewmanEnter email: [email protected] Newman ([email protected]) was inserted as ID 1
Be aware that no validation is being done in this program to make sure a valid SQL statement is being submitted toSQLite. The apostrophe or single quote character is the most common cause of problems, as you can see if youattempt to enter a contact name containing an apostrophe.
[chris@python c]$ ./listing6.3Enter first name: PaddyEnter last name: O'BrienEnter email: [email protected] error: unrecognized token: "@"
INSERT INTO CONTACTS (first_name, last_name, email)
The SQL query displayed in the error message shows the problem caused by the apostrophe. In this case the @symbol causes an error, which appears strange at first but look at the pairs of single quotes in the VALUES list.Before SQLite even attempts to perform the insert, it reads three strings from the line'Paddy', 'O' and ', '. The @ signis not contained in quotes and causes a parser error.
Fortunately the library provides an easy way around this using sqlite_mprintf().
This function works in a similar way to sprintf() except that a pointer to a string buffer is returnedthe memory assignedto the buffer by malloc() should be freed using sqlite_freemem() when it is no longer needed. Using sqlite_mprintf(),the format code %q works like %s, but any single quotes are delimited, so they can be inserted as literal characters.
We could rewrite the statement that builds the SQL query in Listing 6.3 like this:
char *qry;
...
qry = sqlite_mprintf("INSERT INTO CONTACTS (first_name, last_name, email) \n"
For convenience, the function sqlite_exec_printf() works like sqlite_exec() but accepts a printf-style format string withits arguments given after the usual five parameters. Again the %q format code can be used to make sure quotes aredelimited correctly.
We can use this function to combine the sprintf() and sqlite_exec() calls in Listing 6.3 while also protecting againstproblems with single quotes, as the following example demonstrates.
ret = sqlite_exec_printf(db,
"INSERT INTO CONTACTS (first_name, last_name, email) \n"
"VALUES ('%q', '%q', '%q')", NULL, NULL, &errmsg,
first_name, last_name, email);
Replace lines 33 to 36 with the preceding statement, and then recompile and execute the program to attempt to insertthe contact name that caused us problems before:
The INSERT statement in the preceding example used an implicit transaction to execute a single change to thedatabase. Where an explicit transaction is required around a group of statements, it can be started simply by executinga BEGIN TRANSACTION command through the SQLite C Library.
sqlite_exec(db, "BEGIN TRANSACTION");
Issuing a COMMIT TRANSACTION or ROLLBACK TRANSACTION instruction is done the same way. Beaware that if sqlite_close() is called during an open transaction, an implicit ROLLBACK will take place.
Whenever an UPDATE or DELETE operation is performed, records in the database will be changedassuming ofcourse that the WHERE clause does find some rows that should be affected. The function sqlite_changes() allows usto see just how many matching rows were changed or removed as a result of an SQL command.
Listing 6.4 issues an UPDATE statement converting each first_name and last_name value to uppercase in thecontacts table and displays the number of rows affected by this command. Because there is no WHERE clause in theSQL statement, the number of rows affected will be the total number of rows in the contacts table.
printf("%d row(s) were changed\n", sqlite_changes(db));
}
sqlite_close(db);
}
When we run this program, the response tells us how many rows were affected:
$ ./listing6.42 row(s) were changed
Note
The number returned by sqlite_changes() is the total number of rows that could be altered by the UPDATEregardless of whether or not the new value is different from the old value. Therefore if this program is run a secondtime SQLite still reports the same number of changes, even though the data effectively remains the same.
Callback Functions for SELECT Queries
In order to process the results of a query from sqlite_exec(), a callback function is used that is invoked once for eachrow in the dataset. The callback function must have this prototype:
int callback(void *pArg, int argc, char **argv, char **columnNames)
The pArg parameter to the callback function is a copy of pArg from the sqlite_exec() call. This can be used to passarbitrary data to the callback, but may be NULL if not required.
The value of argc is the number of columns returned by the query. The array argv contains pointers to strings for eachresultthe size of the array is argc. The fourth parameter is also an array in which the first argc entries are the columnnames from the query.
If you turn on the SHOW_DATATYPES pragma, the columnNames array will also contain the column types fromthe database schema, in the next argc elements onwards. The PRAGMA command is discussed in detail in Chapter10, "General Database Administration," but for now this particular setting is turned on by issuing the followingstatement.
PRAGMA SHOW_DATATYPES=ON;
Listing 6.5 contains how a callback function is used to output the entire contents of a dataset.
Listing 6.5. Displaying the Results of a SELECT Query Using a Callback Function
#include <stdio.h>
#include <sqlite.h>
int callback(void *pArg, int argc, char **argv, char **columnNames)
fprintf(stderr, "Could not open database: %s\n", errmsg);
sqlite_freemem(errmsg);
exit(1);
}
ret = sqlite_exec(db, "PRAGMA SHOW_DATATYPES=ON", NULL, NULL, NULL);
ret = sqlite_exec(db, "SELECT * FROM contacts", callback, NULL, &errmsg);
if (ret != SQLITE_OK)
{
fprintf(stderr, "SQL error: %s\n", errmsg);
}
sqlite_close(db);
}
Let's take a closer look at the callback function. We specify callback as the third parameter to sqlite_exec() on line32, which causes the function callback() defined in lines 4 to 12 to be invoked for each row returned by the query.
for (i=0; i<argc; i++) {
printf("%-10s %-8s %s\n",
columnNames[i], columnNames[i+argc], argv[i]);
}
The value in argc is the number of columns returned by the query. In the query from Listing 6.5 this is actually 4thetotal number of columns in contacts. This loop displays one line for each column, using a format string to printf() tospecify a fixed-width column output. We show each column's name (columnNames[i]), type (columnNames[i+argc]),and value (argv[i]).
The callback function ends with
return(0);
The return value from the callback function is importanta non-zero value returned from callback() will cause the callingsqlite_exec() function to return an error.
Compiling and executing this program outputs the contents of the database, as shown here:
Historically, if you wanted to retrieve the results of an SQL query using the C interface, a callback function was theonly method. However, most programmers found this way of coding inconvenient, so a new method becameavailable.
The new method allows you to work with SQLite at a lower level by manipulating a SQLite virtual machine. It usesthree functions to replace sqlite_exec(), as follows.
Compile a single SQL statement with sqlite_compile()
Execute sqlite_step() once for each row of output
Call sqlite_finalize() to clean up when execution has finished
Compilation is the process that takes an SQL statement and generates a SQLite virtual machine capable of runningthat statement. The workings of the Virtual Database Engine (VDBE) are discussed in Chapter 10.
The prototype for sqlite_compile() is
int sqlite_compile(
sqlite *db,
const char *zSql,
const char **pzTail,
sqlite_vm **ppVm,
char **pzErrmsg
);
As before, the db is an open database resource pointer and the second parameteridentified here as zSqlis the SQLstatement itself. The return value is SQLITE_OK if compilation succeeded; otherwise, another error code is returnedand pzErrmsg will point to a textual error message.
The behavior of sqlite_compile() differs from sqlite_exec() in that only one SQL statement can be compiled at a time.If two or more statements are passed in zSql, the third parameter pzTail will point to the first character following theend of the compiled statement so that any remaining SQL commands can be handled.
After successful compilation, the pointer passed as the fourth parameter will point to the virtual machine, stored in anopaque sqlite_vm type structure. With a valid virtual machine pointer, sqlite_step() can be called once or more timesto return a single row of results. This is the prototype for sqlite_step():
int sqlite_step (
sqlite_vm *pVm,
int *pN,
const char ***pazValue,
const char ***pazColName
);
The pVm parameter should be a valid pointer to a virtual machine created with sqlite_compile(). The number ofcolumns in the result set is pointed to by pN. The pazValue and pazColName arguments each point to an array ofpointers, which are used to reference the column values and their names and data types respectively.
Note
The data type information is always included in pazColName along with the column names when using sqlite_step()regardless of whether the SHOW_DATATYPES pragma has been turned on.
The return value from each sqlite_step() call is an integer code from a subset of those shown in Table 6.1. Only thefollowing codes will be used:
SQLITE_BUSY
SQLITE_ROW
SQLITE_DONE
SQLITE_ERROR
SQLITE_MISUSE
A SQLITE_BUSY code indicates that the database file is locked by another thread or process and the callingfunction should handle this. Sleeping for a short amount of time to wait for the lock to clear is an acceptable action totake, after which sqlite_step() can be called again. The function sqlite_busy_timeout can be used to specify a numberof milliseconds for which SQLite should wait for a lock to clear before returning SQLITE_BUSY.
Note
A custom handler for locked databases can be registered with the sqlite_busy_handler() instruction, to specify afunction that will be executed whenever SQLite attempts to access a busy database file. A non-zero return code willcause the interface to attempt to open the file again and the cycle continues until the handler function returns zero.
If a row from the data set has been made available in pazValue and pazColumn, the return code will beSQLITE_ROW. If the last row has already been retrieved, a subsequent call will return SQLITE_DONE, sosqlite_step() will be called in total once more than the actual number of rows in the data set.
Should an error occur, the SQLITE_ERROR code is returned. After either SQLITE_DONE or SQLITE_ERRORit is a misuse of the library to attempt to invoke sqlite_step() a further time, and doing so will generate theSQLITE_MISUSE return code.
Note
The SQLITE_MISUSE error code is intended as a debugging aid, but because of the nature of the problems itattempts to detect, you should not rely on it. In some instances such misuse of the library may result in a programcrash.
When all processing is complete, sqlite_finalize() should be called to clean up any memory allocated to the virtualmachine. This function can also be used to find details of an error that caused sqlite_step() to return aSQLITE_ERROR. Here's the prototype:
int sqlite_finalize(
sqlite_vm *pVm,
char **pzErrmsg
);
The return code of sqlite_finalize() indicates the overall success or failure of the SQL statement and is the same codethat would have been returned by sqlite_exec() if the same statement had been executed using that method.
Though it is a requirement to call sqlite_finalize() when either SQLITE_DONE or SQLITE_ERROR is returned, thevirtual machine can be deleted at an earlier stage. A premature sqlite_finalize() instruction will interrupt processingimmediately and return the database to its previous state.
Listing 6.6 shows how the same query we executed using the callback method in Listing 6.5 can be done with thenon-callback interface.
Listing 6.6. Using the Non-Callback Interface to Execute a Query
The sqlite_get_table() function is provided for convenience and acts as a wrapper around the core API functions thatwe have already seen. This function executes an SQL query and returns a pointer to a data structure that contains theentire result at once.
The prototype for sqlite_get_table() is as follows:
int sqlite_get_table(
sqlite *db,
char *sql,
char ***result,
int *nrow,
int *ncolumn,
char **errmsg
);
In addition to the usual db and sql parameters, the function also requires passing four pointers of the types shown. Theresult created is a one-dimensional array of character pointers, and so in order to work out which elementcorresponds to which data item, the nrow and ncolumn values reference the total number of rows and columnsreturned respectively.
The result array contains one element for each column of every row in the dataset, plus one additional element foreach of the column headings. The column headings are stored first, in result[0] to result[ncol-1]. The value of the firstcolumn in the first row returned will be result[ncol], and so on.
The example in Listing 6.7 performs a simple loop to output the contents of the contacts table usingsqlite_get_table().
Listing 6.7. Using sqlite_get_table to Fetch an Entire Dataset at One Time
When you are done with the result of a call to sqlite_get_table(), the memory allocated should be freed usingsqlite_free_table(result).
Another function combines the convenience of sqlite_get_table() with sqlite_mprintf() to enable automatic delimitingof single quote characters using the %q format code. sqlite_get_table_printf() takes the same first six arguments assqlite_get_table(), with the ability to use a printf-style format string in the sql parameter, which uses values from theseventh argument onwards.
Adding New SQL Functions The SQLite library allows the SQL language to be extended with new functions implemented as C code. In fact all ofSQLite's built-in functions are now written this way.
Creating a regular function begins with a call to sqlite_create_function(), and this is the prototype:
int sqlite_create_function() {
sqlite *db,
const char *zName,
int nArg,
void (*xFunc)(sqlite_func *, int, const char **),
void *pUserData
);
The example in Listing 6.8 defines a trivial function in C that capitalizes every alternate letter in a string. The exampleuses sqlite_create_function() to register the function defined in capitalize_alternate() for use within SQLite under thename altcaps().
Listing 6.8. Creating a Custom Function Using sqlite_create_function()
#include <stdio.h>
#include <sqlite.h>
void capitalize_alternate(sqlite_func *context, int argc, const char **argv)
ret = sqlite_get_table(db, "SELECT altcaps('this is a test')",
&result, &rows, &cols, &errmsg);
if (ret != SQLITE_OK)
{
fprintf(stderr, "SQL error: %s\n", errmsg);
}
else {
for (i=0; i< cols * (rows+1); i++) {
printf("result[%d] = %s\n", i, result[i]);
}
}
sqlite_close(db);
}
First of all we declare the function capitalize_alternate() in C, which will do the work we want to include in SQL.
The prototype for a function that can be used with sqlite_create_function() is fixed; the context argument can be usedto pass arbitrary data to the function. The argc and argv arguments tell the C function how many parameters havebeen passed when it is called from SQL.
void capitalize_alternate(sqlite_func *context, int argc, const char **argv)
The loop cycles through each item in the string passed inargv[0] will contain the only parameter in this caseand usestoupper() and tolower() in turn to alternate the case of each character, adding to str one character at a time.
for (i=0; i<strlen(argv[0]); i++) {
if (i%2 == 0)
str[i] = toupper(argv[0][i]);
else
str[i] = tolower(argv[0][i]);
}
str[i] = '\0';
Finally, we tell the interface that str is our return value using sqlite_set_result_string():
sqlite_set_result_string(context, str, -1);
Three different functions enable return values of different data types to be used:
The third int parameter to sqlite_set_result_string() and sqlite_set_result_error() is the number of characters to takefrom the string. If the value is negative, as in our example, all characters up to and including the first \0 are returned.
So with the custom function defined in C, a call is made to sqlite_create_function() to add it to the SQL language.
The zName parameter here tells SQLite that the function is called altcaps(). The name used in SQL does not have tomatch the C function name, although it is possible to do so.
To test that the function has been registered correctly, sqlite_get_table() is called to execute a simple query usingaltcaps().
ret = sqlite_get_table(db, "SELECT altcaps('this is a test')",
&result, &rows, &cols, &errmsg);
If the query is successful, the contents of the returned array are displayed:
for (i=0; i< cols * (rows+1); i++) {
printf("result[%d] = %s\n", i, result[i]);
Running the compiled program will produce the following output, showing that the function has indeed capitalizedevery other letter of the string.
$ ./listing6.8result[0] = altcaps('this is a test')
result[1] = ThIs iS A TeSt
Note
Although sqlite_create_function() takes a database resource parameter and registers the function only to thatdatabase, the function is not stored in the database file. Therefore when the program that calls sqlite_create_function()has terminated, the user-defined function is no longer available to SQLite.
If required within a function's code, the data contained in the pointer pUserData in the sqlite_create_function() call canbe referenced using the sqlite_user_data() function, which has this prototype:
void *sqlite_user_data(sqlite_func *)
Creating Aggregating Functions
The process to create an aggregating functionone that applies a calculation across all the rows returned by a query oracross similar rows indicated by a GROUP BY clauseis slightly different from a regular function.
The interface call sqlite_create_aggregate() takes two function pointer arguments. One function is called for each rowin the result setthe step functionand one is called once all the rows have been processedthe finalize function. Theprototype is as follows:
To see an example, let's take a look at how SQLite actually implements the sum() aggregating function. The sourcecode for this and many other built-in functions can be found in func.c in the SQLite source distribution.
The step function sumStep() is as follows:
static void sumStep(sqlite_func *context, int argc, const char **argv){
SumCtx *p;
if( argc<1 ) return;
p = sqlite_aggregate_context(context, sizeof(*p));
if( p && argv[0] ){
p->sum += sqliteAtoF(argv[0], 0);
p->cnt++;
}
}
SumCtx is a data structure suitable for storing the values needed to compute the return value; in this case a runningtotal is added to each step to produce the sum. In fact because the avg() function is computed from the sum dividedby the number of elements, the SumCtx structure also includes a running count of the number of terms and is used forboth functions.
typedef struct SumCtx SumCtx;
struct SumCtx {
double sum; /* Sum of terms */
int cnt; /* Number of elements summed */
};
The sqlite_aggregate_context() function provides access to an area of memory that is available only to the particularinstance of a function called from SQL, and the memory is automatically freed after the related finalize function hasbeen called.
The work done by the step function for sum() is actually quite simple. The running total is incremented by the currentvalue and the running count is incremented by one:
p->sum += sqliteAtoF(argv[0], 0);
p->cnt++;
The function sqliteAtoF() is a SQLite-safe version of the regular atof() function to convert an ASCII string to afloating-point number and is defined in util.c. If the locale setting requires it, sqliteAtoF() will accept a comma as adecimal point instead of a period character.
The sumFinalize() function references the context structure containing the totals computed in the step function anduses sqlite_set_result_double() to return a floating-point result containing the sum of all the values:
static void sumFinalize(sqlite_func *context){
SumCtx *p;
p = sqlite_aggregate_context(context, sizeof(*p));
sqlite_set_result_double(context, p ? p->sum : 0.0);
}
We mentioned before that the avg() function can reuse much of the code for the sum() function, and in fact the samestep function is used to create avg() in SQLite. The following code shows the avgFinalize() function.
static void avgFinalize(sqlite_func *context){
SumCtx *p;
p = sqlite_aggregate_context(context, sizeof(*p));
Chapter 7. The Perl Interface In this chapter we will look at the Perl interface to SQLite. Databases are accessed from Perl scripts using theDatabase Interface (DBI) and a Database Driver (DBD) for SQLite. If you have used DBI/DBD to interface Perlwith other database systems, much of the procedure to communicate with SQLite will be familiar to you.
This chapter gives an overview of Perl's DBI in general and also details the attributes and methods specific to theSQLite DBD.
Preparing to Use the SQLite Interface To access a SQLite database from Perl you need to install both the DBI module and the DBD module for SQLite.
If you do not already have DBI installed, use cpan to download it from the Comprehensive Perl Archive Networkand install it.
# cpancpan> install DBI
On Windows platforms using the ActivePerl distribution, use the ppm.bat script to install Perl modules.
C:\perl\bin> ppm.batppm> install DBI
Note that it can take a while to download, configure, compile, and install CPAN modules. If you are accessingCPAN for the first time, a set of modules will also be downloaded first to update your system.
To add the SQLite DBD module to your system, install DBD::SQLite2 from CPAN.
# cpancpan> install DBD::SQLite2
The DBD::SQLite2 package includes as much of SQLite as it needs, so there is no need to have the SQLite librarieson your system before installing the DBD moduleyou can add SQLite support to Perl on a fresh system withoutneeding to download anything from sqlite.org, although you will probably want to install the sqlite monitor tool too.
Note
As we have chosen to stick with the existing, stable, and well-supported SQLite 2 engine throughout this book, youshould install DBD::SQLite2 for a compatible Perl database driver. The original DBD::SQLite module uses the latestSQLite libraryversion 3which uses a different database file format than the previous version.
Though the Perl DBI abstracts the actual database back end and the examples in this chapter will work withwhatever SQLite version you use, you also need to be using the appropriate sqlite tool for your SQLite library versionin order to read your databases. You can read about the changes in SQLite 3 in Appendix I, "The Future of SQLite."
About the Perl DBI Perl's Database Interface uses a database abstraction model, so it doesn't really matter what the underlying databaseis. The DBI module calls the appropriate Database Driver module and passes off the SQL commands for execution.
There are many more DBD modules than you are ever likely to use, supporting all the major database systems andmany of the minor ones. The naming convention is DBD::dbname, for example DBD::Oracle for Oracle orDBD::Sybase for Sybase (and SQL Server). There's also DBD::ODBC for other ODBC-enabled databases thatdon't have their own specific DBD module and even DBD::CSV to interact with comma-separated-values files usingDBI.
The DBD module we are interested in is called DBD::SQLite2, and the way in which queries from your Perl scriptinteract with DBI and the SQLite DBD driver is shown in Figure 7.1.
Figure 7.1. The Perl Database Interface model.
The DBI module is loaded into your script with this simple command:
use DBI;
You do not need to use any specific DBD module as DBI will take care of that when it is needed. This makes it easyto write highly portable database applications in Perl because only a single instruction needs to be changed to tell DBIto work with a different database. The instruction looks like this for SQLite:
The prototype for DBI->connect() has three parametersa data source, username, and password. As SQLite does notuse user-based authentication, the second and third parameters are always blank.
The data source contains three parts separated by colons: the keyword DBI, the database typein this caseSQLiteand an expression indicating the name of the database. The filename given as dbfile can include an absolute orrelative path or will be opened from the current working directory if no path is given.
Getting Information About the DBI
The DBI module will report back its supported DBD drivers through the available_drivers() method. The script in Listing 7.1 grabs the available drivers as an array and loops through to print each entry to screen.
Listing 7.1. Using available_drivers() to Check What DBD Modules Are Installed
#!/usr/bin/perl -w
use DBI;
use strict;
my @drivers = DBI->available_drivers();
foreach my $driver (@drivers) {
print "$driver \n";
}
The output will look similar to this:
# ./listing7.1.plCSV
DBM
ExampleP
File
Proxy
SQLite
Sponge
Your output may be different if you have additional DBD modules already installed on your system. Of those shown inthe preceding example, only CSV, DBM, and SQLite are actual database drivers. ExampleP is a simple example of adriver that can be used to help write new DBD drivers. The others are drivers that perform internal operations sharedby other DBD drivers.
DBI also provides the installed_versions method, which will give a formatted report of both the supported DBDmodules and their installed versions as well as details of the Perl and operating-system versions. This method caneasily be called from the command line as follows:
Let's look at how the same query can be performed using two different DBD modules by simply changing theDBI->connect() instruction. Listing 7.2 shows a script that will connect to a SQLite database, create a very simpletable, and insert a few rows of data.
Listing 7.2. Creating Some Simple Data Records Using Perl DBI
#!/usr/bin/perl -w
use strict;
use DBI;
my $dbh = DBI->connect("DBI:SQLite2:dbname=perldb", "", "");
my $sql = "CREATE TABLE mytable (mynumber INTEGER)";
$dbh->do($sql);
for (my $i=1; $i<=5; $i++) {
$sql = "INSERT INTO mytable VALUES ($i)";
$dbh->do($sql);
}
The DBI->connect() call opens up a database called perldb from the current working directory. Unless you alreadyhave a database by this name in that directory, a new file will be created.
To send an SQL statement to be processed by DBI, we use the do() method on a database handle.
$dbh->do($sql);
Running the script in Listing 7.2 will create a new database file called perldb (unless one already existed, of course),and we can view the schema of the new table and view the rows that were inserted.
Now let's try the same thing using a different DBD module. One of the simplest, although crudest, ways to store datato the filesystem is in a comma-separated-values file, and Perl can use a DBD module to communicate with CSV filesusing an SQL interface. If your system does not already have it, install the DBD::CSV module using cpan.
cpan> install DBD::CSV
To adapt the script from Listing 7.2 to save data to comma-separated files rather than SQLite, simply change theDBI->connect() instruction so that it reads as follows:
The data source for the CSV driver has an optional assignment, f_dir, for the output directory and uses the currentdirectory if this is not given. One CSV file is written per table, rather than one file for the entire database in SQLite.The operation is successful in creating a file called mytable containing the same data rows as before, as you can seeby examining the new file created in /tmp.
$ cat /tmp/mytablemynumber
1
2
3
4
5
Though the way the data files are written is different with the CSV driver, the same SQL code that we wrote to createtables in SQLite will also create a CSV if the data source is changed. The same procedure can be used for anysupported DBD module, whether it is a filesystem-based database or a client/server RDBMS.
For the remainder of this chapter we will look at examples with DBD::SQLite2, but do remember that the Perlinterface for SQLite is not specific to SQLite. Indeed you may already have an existing Perl application that uses DBIwith a different database that could easily be ported to SQLite by simply changing the DBD module in the datasource.
Using the SQLite DBD In this section we will look at the methods and attributes available for a DBI object with examples specific to theDBD::SQLite2 module.
Opening and Closing the Database
As we saw before, the DBI->connect() function is used to open a database, and for a SQLite database no usernameor password arguments are required. The usage is always as follows:
Some basic error trapping is useful in case the connection to the database fails. Because SQLite will create a newdatabase file with the given name if one does not exist, DBI->connect() will only fail if there is an I/O error, forinstance file permissions not allowing the file to be opened or created in a particular directory, or no more disk spaceon the device.
The errstr property contains the most recent database error. The following example could be used to exit with anerror message if SQLite is unable to open the specified dbfile:
To close the connection to a database, simply invoke the disconnect() method on your DBI object.
$dbh->disconnect();
Executing SQL Statements
We have already seen that the do() method can be used to pass SQL to the DBI module in order to execute acommand. In fact, do() is good only for non-SELECT statements and the preferred method to send SQL commandsto the DBI module is using a two-step process. do() is a convenience function that effectively combines these twosteps.
First the query needs to be prepared by the SQL engine using prepare, after which it can be executed using theexecute method.
Listing 7.3 creates the contacts table in SQLite with error trapping at every stage. The errstr property returns themost recent error message generated by the DBD, so any cause of error that causes the script to exit prematurely willbe displayed to the screen.
Listing 7.3. Creating a Table Using DBI
#!/usr/bin/perl -w
use strict;
use DBI;
my $dbh = DBI->connect("DBI:SQLite2:dbname=perldb", "", "")
or die("Cannot connect: " . DBI::errstr() );
my $sql = "CREATE TABLE contacts ( ".
" id INTEGER PRIMARY KEY, ".
" first_name TEXT, ".
" last_name TEXT, ".
" email TEXT) ";
my $sth = $dbh->prepare($sql)
or die("Cannot prepare: " . DBI::errstr() );
my $ret = $sth->execute()
or die("Cannot execute: " . DBI::errstr() );
The prepare() method returns a statement handle object, which in turn can invoke the execute() method to performthe query.
$sth = $dbh->prepare($sql);
...
$ret = $sth->execute();
Note
The prepare() method instructs SQLite to get ready to execute the statementthe database engine will tokenize thestatement and work out how it will be executed when the time comesbut it does not parse the SQL statement. Anysyntax errors in the SQL will only be reported when execute() is called.
Using Bind Variables
A powerful feature of DBI is that an SQL statement still needs to be prepared only once, even if it is to be executedrepeatedly with different values in the statement. This is achieved using placeholders in the SQL and an array of bindvalues passed as an argument to execute() and gives a significant performance saving over preparing a query withstatic values each time it is executed.
A placeholder is indicated by the question mark symbol. For example, the following query could be prepared tocreate an SQL INSERT statement where the placeholders indicate that we will supply the values to be inserted atexecution time. Note that no quotes are required around a placeholder even if a string value is the expectedsubstitution.
INSERT INTO contacts (first_name, last_name, email) VALUES (?, ?, ?)
Listing 7.4 shows how a record is inserted using this technique.
Listing 7.4. Inserting a Record Using Bind Values
#!/usr/bin/perl -w
use strict;
use DBI;
my $dbh = DBI->connect("DBI:SQLite2:dbname=perldb", "", "")
or die("Cannot connect: " . DBI::errstr() );
my $sql = "INSERT INTO contacts (first_name, last_name, email) ".
The execute() method takes one argument for each placeholder in the order specified in the query. We can verify thatthe record was inserted as expected using the sqlite tool.
$ sqlite perldbSQLite version 2.8.15
Enter ".help" for instructions
sqlite> SELECT * FROM contacts;id first_name last_name email
Let's extend this a little further to create a script that allows you to populate the contacts table by entering recordsfrom the keyboard.
The script in Listing 7.5 creates a loop that reads user input that is expected to be the three elements first_name,last_name, and email separated by commas. If the format of the input is valid, the record is inserted into the database;otherwise, the script will exit.
Listing 7.5. Inserting Records from User Input with a Single Prepared SQL Statement
#!/usr/bin/perl -w
use strict;
use DBI;
my $dbh = DBI->connect("DBI:SQLite2:dbname=perldb", "", "")
or die("Cannot connect: " . DBI::errstr);
my $sql = "INSERT INTO contacts (first_name, last_name, email) ".
The key part of this script is the while loop, which reads from standard input. Input is taken from stdin a line at a time,and we use chomp to strip the trailing newline characters.
while(<>) {
chomp;
The expected data format is a comma-separated list of the three elements to be inserted into contacts. The splitinstruction breaks up the read line of input and the first three comma-separated elements are assigned to $first_name,$last_name, and $email respectively.
my ($first_name, $last_name, $email) = split /,/;
To validate the data record we test that a value has been assigned to $email. If there are less than threecomma-separated values in the input, $email will be empty. If there are more than three elements, the fourth andsubsequent elements are simply discarded.
If everything is okay, a row is inserted by calling the execute() method on the prepared SQL statement with theassigned variables.
$sth->execute($first_name, $last_name, $email)
or die("Cannot execute: ". DBI::errstr );
Note
The placeholders in an SQL statement can only represent items that do not alter the execution plan. Therefore youcannot use a placeholder to substitute a table or column name.
The bind_param() function is used to set bind parameter values individually and provides a little more functionalitythan supplying a list of bind values to the execute() method by allowing you to give a data type hint.
In fact when bind parameters are passed to the execute() method, the DBI effectively calls bind_param() for eachvalue in the list. The data type is assumed to be SQL_VARCHAR.
The syntax of bind_param() is as follows:
$sth->bind_param($p_num, $bind_value, $bind_type)
The parameters are numbered in the order they appear in the SQL starting at 1, and the number of the parameter youwant to assign is passed in the $p_num argument.
The only data type constants appropriate for SQLite are SQL_VARCHAR and SQL_INTEGER. In order to usethese constants in your script you must import them with the use command:
use DBI qw(:sql_types);
To produce the same result we saw in Listing 7.4 using bind_param(), instead of passing the bind values to execute(),the following lines would be required:
$sth->bind_param(1, $first_name, SQL_VARCHAR);
$sth->bind_param(2, $last_name, SQL_VARCHAR);
$sth->bind_param(3, $email, SQL_VARCHAR);
$sth->execute() or die "Cannot execute: ". DBI::errstr;
Bind values are safe when passed into an SQL statement. For example, the following execution of Listing 7.5 showsthat a surname containing an apostrophe does not conflict with the implied quotation marks around that string value.
Had we performed the same INSERT command using static values in the script, some kind of delimiting would berequired to avoid the query looking like this:
INSERT INTO contacts (first_name, last_name, email)
This query would cause an error upon execution as the parser would not be able to determine that the apostrophe inO'Brien was not in fact a closing quotation mark around the string 'O'.
In Perl we can call the DBI method quote() to put a value into the right type of quotes for the underlying database(single quote for DBD::SQLite2) while adding the necessary delimiters to make them safe for use within an SQLcommand. Listing 7.6 performs the quote method on each of the values in an INSERT statement.
Listing 7.6. Using the quote Method to Safely Delimit String Values
#!/usr/bin/perl -w
use strict;
use DBI;
my $dbh = DBI->connect("DBI:SQLite2:dbname=perldb", "", "")
or die("Cannot connect: " . DBI::errstr() );
my $first_name = "Paddy";
my $last_name = "O'Brien";
my $email = "paddy\@irish.com";
my $sql = sprintf("INSERT INTO contacts (first_name, last_name, email) \n".
"VALUES (%s, %s, %s)",
$dbh->quote($first_name),
$dbh->quote($last_name),
$dbh->quote($email));
print $sql . "\n";
The output from running Listing 7.6 displays the SQLite-safe query text.
$ ./listing7.6.plINSERT INTO contacts (first_name, last_name, email)
Remember that SQLite delimits the apostrophe character by doubling itthe occurrence of '' in a string enclosed bysingle quotes means that the string contains one apostrophe.
Note
Because quote() encloses the string in quotes as well as delimiting any awkward characters, it should not be usedwith placeholders and bind values. As we have already seen, apostrophes in bind values do not need to be delimited.
Using Commands That Change the Database
When an UPDATE or DELETE operation is performed, it is usually the intention to alter one or more databaserows. The number of rows affected by the statement can be found by looking at the rows property of the statementhandler.
$sth->execute();
print $sth->rows;
The number of affected rows is also the return value from execute, which provides a handy shortcut to thisinformation. Because the do() method is a convenience function that prepares and executes an SQL command in onego, you can also look at the return value from do() to find the number of affected rows.
For example, after the following statement is executed, $num will contain the number of rows that has been deleted:
$num = $dbh->do("DELETE FROM contacts WHERE first_name = 'Chris'");
When an INSERT is performed on a table with an INTEGER PRIMARY KEY fieldas in our previousexamplesSQLite assigns the next numerical value to that field if the inserted value is NULL. The SQLite DBD moduleextends DBI to provide a private method for retrieving this value. It is called as follows:
$id = $dbh->func("last_insert_rowid").
Note
The INTEGER PRIMARY KEY property of a database column is specific to SQLite, so last_insert_rowid isimplemented by DBD::SQLite2, not the DBI module. Therefore, be aware that scripts that use this private methodcannot be expected to work with other DBD modules.
Listing 7.7 performs an UPDATE followed by a INSERT on the contacts table, and displays the number of affectedrows and the assigned INTEGER PRIMARY KEY field respectively.
Listing 7.7. Finding Affected Rows and the Assigned Value of a Primary Key
#!/usr/bin/perl -w
use strict;
use DBI;
my $dbh = DBI->connect("DBI:SQLite2:dbname=perldb", "", "")
or die("Cannot connect: " . DBI::errstr() );
my $sql = "UPDATE contacts ".
"SET first_name = upper(first_name)";
my $sth = $dbh->prepare($sql)
or die("Cannot prepare: " . DBI::errstr() );
my $num = $sth->execute or die("Cannot execute: ". DBI::errstr() );
print "Updated " . $sth->rows ." rows \n";
$sql = "INSERT INTO contacts (first_name, last_name, email) ".
$sth->execute() or die("Cannot execute: ". DBI::errstr() );
print "Inserted with ID " . $dbh->func("last_insert_rowid"). "\n";
The following output will be generated, depending on the actual number of records already in the contacts table:
$ ./listing7.7.plUpdated 4 rows
Inserted with ID 5
Transactions
Three DBI functions are provided to begin and end SQLite transactions. They map directly to the underlying SQLcommands and each is equivalent to issuing the corresponding command shown in Table 7.1.
Table 7.1. Transaction Functions in Perl DBI
begin_work() BEGIN TRANSACTION
commit() COMMIT TRANSACTION
rollback() ROLLBACK TRANSACTION
Fetching Records from the Database
We have already seen how an INSERT or UPDATE command returns the number of affected rows as the result ofthe execute method. Now let's take a look at how the result of a SELECT statement is processed using the Perl DBI.
The fetchrow_array() method can be used on an executed statement handler to return a list of the selected columnvalues. The first time it is called, the first row from the query is returned, and on subsequent calls the record pointer isadvanced to return the next row in sequence.
Listing 7.8 shows how fetchrow_array() can be used in a loop to output every row from a table.
Listing 7.8. Using fetchrow_array() to Output the Contents of a Table
#!/usr/bin/perl -w
use strict;
use DBI;
my $dbh = DBI->connect("DBI:SQLite2:dbname=perldb", "", "")
or die("Cannot connect: " . DBI::errstr() );
my $sql = "SELECT first_name, last_name FROM contacts";
my $sth = $dbh->prepare($sql)
or die("Cannot prepare: " . DBI::errstr() );
$sth->execute() or die("Cannot execute: ". DBI::errstr() );
while( my ($first_name, $last_name) = $sth->fetchrow_array() ) {
print "$first_name $last_name \n";
}
The while loop assigns two variables from the elements of the array returned by fetchrow_array() for each iteration.
while( my ($first_name, $last_name) = $sth->fetchrow_array() ) {
After the last row of data has been fetched, the while condition is false and the loop exits. The output generated by Listing 7.8 is as follows:
$ ./listing7.8.plCHRIS Newman
PADDY O'Brien
BILL Williams
As an alternative to fetchrow_array(), you can use the fetchrow_arrayref() method, which returns a reference to anarray containing each row's elements. A while loop to achieve the same result as Listing 7.8 using this method wouldlook like this:
while (my $ref = $sth->fetchrow_arrayref()) {
print "@{$ref} \n";
}
The data rows can also be returned as a hash, using the fetchrow_hash() method. A reference to a hash is returnedwhere the hash contains column name and value pairs.
while (my $ref = $sth->fetchrow_hashref()) {
print "$$ref{'first_name'} $$ref{'last_name'}\n";
}
Note
Where values in the table are NULL, they are seen in Perl as undef, regardless of which method you choose to fetchthe data.
The functions that begin fetchrow_ also have corresponding functions that begin fetchall_. The difference, as you mightexpect, is that the entire dataset returned from the query is fetched all at once, rather than a row at a time.
Listing 7.9 shows how the result fetched by fetchall_arrayref() can be used in Perl.
Listing 7.9. Using fetchall_arrayref to Fetch an Entire Dataset
#!/usr/bin/perl -w
use strict;
use DBI;
my $dbh = DBI->connect("DBI:SQLite2:dbname=perldb", "", "")
or die("Cannot connect: " . DBI::errstr() );
my $sql = "SELECT * FROM contacts ORDER BY first_name";
my $sth = $dbh->prepare($sql)
or die("Cannot prepare: " . DBI::errstr() );
$sth->execute() or die("Cannot execute: ". DBI::errstr() );
my $ref = $sth->fetchall_arrayref();
foreach my $row (@{$ref}) {
print "@$row\n";
}
The output looks like this, with the columns displayed in the order they appear in the table schema:
An optional argument to fetchall_arrayref() can be used to fetch only a subset of the columns returned by the queryinto the array, known as a slice. The argument should be a hash reference of the column numbers to be returned,where columns are numbered from zero starting with the first in the selected list. Negative numbers can be used tocount from the end of the column list.
For example, the following statement would cause only the second and last columns from the query to be put into thearray:
$ref = $sth->fetchall_arrayref([1,-1]);
An optional second argument allows the maximum number of rows that will be created in the array to be specified. Anon-zero value will restrict the dataset if it is larger than the total number of rows returned by the query, including theLIMIT clause if there is one.
If either argument is not required, it can be specified as undef. The following statement does not slice the array butlimits the number of rows returned to 2:
$ref = $sth->fetchall_arrayref(undef, 2);
The fetchall_hashref() method requires an argument that gives the name of the field to be used as the key of thereturned hash. The key field argument may be the column name or number.
The following statement calls fetchall_hashref() on contacts using the email column as the key:
$ref = $sth->fetchall_hashref('email');
Given an email address, we can now use this hash to find other information:
The finish method that can be called on a statement handle is used to destroy the handle when it is no longer needed.It is simply called as follows:
$sth->finish();
However, finish() should only be called when you are done with reading data from the statement handle and knowthat there is more data still to come. If any of the fetch instructions have come to the end of the dataset, there is noneed to call finish().
Error Checking
The error checking we have come across so far has tested the return code of a function and taken action to displaythe error and exit if necessary. This might look like the following example using a traditional if condition to check forsuccess:
$ret = $sth->execute();
if (!$ret) {
print DBI::rrstr;
exit;
}
Or it can be done in the more concise syntax using or die.
$ret = $sth->execute()
or die("Cannot execute: " . DBI::errstr);
There is an even more concise way to apply error checking to all your database function calls. The Perl DBIimplements automatic error checking using two built-in exceptions, PrintError and RaiseError, enabled by settingattributes with those names against the database handle.
The values of PrintError and RaiseError can be set at the time the database connection is established via the attrargument, using 1 to turn that feature on and 0 to turn it off.
To create a database connection with automatic error checking handled using RaiseError, you would use a statementsimilar to the following:
The PrintError exception causes the warn() function to be called with the error stringthe value of the errstr propertyasits argument. The error message will be displayed, but program execution will not stop.
The RaiseError exception causes the die() function to be called, halting program execution. If both PrintError andRaiseError are enabled, warn() is called first, then die().
The status of each exception can be changed mid-script simply by reassigning the value of that attribute. To turn offRaiseError, for instance, you would do the following:
$dbh->{RaiseError} = 0;
Tracing
The Perl DBI includes a powerful tracing mechanism that allows you to follow a query's execution to aid debugging.
Tracing is enabled using the trace() method, which can be called on a database handle or on the DBI itself to alter thedefault settings for every database handle opened in the script. Its argument is a trace level from 0 to 15; the valuesfor which are shown in Table 7.2.
Table 7.2. Trace Level Values and Their Meanings
0 Trace disabled.
1 Trace DBI method calls returning with results or errors.
2 Trace method entry with parameters and returning withresults.
3 As level 2, with some additional high-level informationfrom the DBD and some internal information from theDBI.
4 As level 3, with more detailed information from the DBD.
515 Each higher level adds more obscure tracing information.
Trace level 1 should give a good overview of what is going on in your script. Level 2 will give some extra informationthat can be useful when debugging, and the higher levels are really only any use if you are trying to trap a specificproblem.
To perform a simple trace of your script using trace level one, use the following command:
DBI->trace(1);
Trace output is written to stderr by default; however, an optional second parameter to trace() allows this to bechanged.
Listing 7.10 creates a new table, numbers, and performs a simple loop to insert the first few square numbers into thenum column. This is a trivial example to show tracing in action, but we will use the data in this listing later in thischapter.
Listing 7.10. Inserting a Series of Square Numbers
#!/usr/bin/perl -w
use strict;
use DBI;
my $dbh = DBI->connect("DBI:SQLite2:dbname=perldb", "", "")
or die("Cannot connect: " . DBI::errstr() );
$dbh->trace(1);
my $sql = "CREATE TABLE numbers (num INTEGER)";
$dbh->do($sql);
$sql = "INSERT INTO numbers (num) VALUES (?)";
my $sth = $dbh->prepare($sql)
or die("Cannot prepare: " . DBI::errstr() );
for (my $i=1; $i<=9; $i++) {
$sth->execute($i * $i)
or die("Cannot execute: " . DBI::errstr() );
}
Tracing is set to level 1 at the DBI level. Running this script produces trace output similar to the following:
$ ./listing7.10.pl DBI::db=HASH(0x9b88b90) trace level set to 0x0/1 (DBI @ 0x0/0)
in DBI 1.43-ithread (pid 2509)
<- do('CREATE TABLE numbers (num INTEGER)')= '0E0' at
listing7.10.pl line 11
<- prepare('INSERT INTO numbers (num) VALUES (?)')= DBI::st=HASH(0x9b8bd0c)
at listing7.10.pl line 15
<- execute(1)= 1 at listing7.10.pl line 19
<- execute(4)= 1 at listing7.10.pl line 19
<- execute(9)= 1 at listing7.10.pl line 19
<- execute(16)= 1 at listing7.10.pl line 19
<- execute(25)= 1 at listing7.10.pl line 19
<- execute(36)= 1 at listing7.10.pl line 19
<- execute(49)= 1 at listing7.10.pl line 19
<- execute(64)= 1 at listing7.10.pl line 19
<- execute(81)= 1 at listing7.10.pl line 19
! <- DESTROY(DBI::db=HASH(9ae4248))= undef during global destruction
Tracing can also be enabled from the shell without needing to amend your scripts using the DBI_TRACE environmentvariable. If DBI_TRACE is assigned a non-zero numeric value it will set the trace level for the DBI in Perl scripts tothat value. If the number is followed by =filename, trace output will be redirected to that file.
For example, to run a script in trace level 2 with output written to TRace.log, the following would work in the Bourneshell:
$ DBI_TRACE=2=trace.log perl myscript.pl
Adding New SQL Functions A powerful feature of the SQLite library is the ability to add user-defined functions to the SQL language. Becausethis feature is specific to SQLite, it is not part of the Perl DBI.
Instead, Perl provides the facility to create user-defined functions through two private methods in the SQLite DBD.
Creating Functions
You can register a new function using the private method create_function, called as follows:
The three arguments required are the function name, the number of arguments the function will take, and a referenceto the Perl function that is to be used whenever the SQL function is called.
The simplest way to see how a user-defined function is used is to register a built-in Perl function in the SQL language.The following statement uses a minimal inline function, such as $func_ref, which registers the Perl function rand() asthe SQL function rand(). No arguments are required for the function, so $argc is zero.
$dbh->func( "rand", 0, sub { return rand() }, "create_function" );
SQLite's built-in random() function returns a random signed 32-bit integer, whereas Perl's rand() function returns adecimal between 0 and 1. You could see the difference by preparing and executing the following statement from aPerl script after the rand() function has been registered in SQL:
SELECT rand(), random();
User-defined functions have to be registered for each database connection where they are to be used. Such functionsare available in SQL only for the duration of that connectionthey are not saved to the database itself or made availableto other connection objects within the same script. Of course, user-defined functions are designed to allow you to addcustom functions to SQL, so let's look at a more complex example (see Listing 7.11).
Listing 7.11. Creating a User-Defined Function in Perl
#!/usr/bin/perl -w
sub altcaps {
my $str = $_[0];
my $newstr = "";
for ($i=0; $i<length($str); $i++) {
$char = substr($str, $i, 1);
if ($i%2) {
$newstr .= uc($char);
}
else {
$newstr .= lc($char);
}
}
return $newstr;
}
use strict;
use DBI;
my $dbh = DBI->connect("DBI:SQLite2:dbname=perldb", "", "")
my $sql = "SELECT altcaps(last_name) FROM contacts";
my $sth = $dbh->prepare($sql)
or die("Cannot prepare: " . DBI::errstr() );
$sth->execute() or die("Cannot execute: ". DBI::errstr() );
while ( my $first_name = $sth->fetchrow_array() ) {
print "$first_name \n";
}
Listing 7.11 creates a new function in Perl called altcaps(), which converts the case of a string so that it containsalternating upper- and lowercase characters. The function is then registered as an SQL function with the same nameand is used to select a modified version of the last_name column from contacts.
The output from running this script is as follows:
nEwMaN
o'bRiEn
wIlLiAmS
In this example we specified a $argc value of 1 as only a single string argument is required. SQLite will return an errorcode if the function is passed too many or too few arguments.
For example, if the SQL statement is changed to
$sql = "SELECT altcaps(first_name, last_name) FROM contacts";
The following error is produced:
Cannot execute: wrong number of arguments to function altcaps()
at ./func.pl line 30.
However, you can call the user-defined function as many times as you like in the same query. For example, this is avalid usage of the custom function:
$sql = "SELECT altcaps(first_name), altcaps(last_name) FROM contacts";
Note
If $argc is 1, any number of arguments can be passed to the function. This simply disables validation of the number ofarguments in the SQL parser; you must still deal with the passed-in arguments in a sensible way in your Perl function.
Creating Aggregating Functions
SQLite also allows you to create new aggregating functions through the Perl interface. An aggregating function is onethat applies a calculation across all the rows returned by a query or across similar rows indicated by a GROUP BYclause.
The private method in DBD::SQLite2 that is used to create an aggregating function is called create_aggregate.
This is called in much the same way as create_function, but the $pkg argument is a reference to a package containingthe steps necessary to implement an aggregating function.
Three steps are required: The new() method is called once to initialize an aggregator object upon which the step() and finalize()
methods will be called.
The step() method is called once for each row in the aggregate.
The finalize() method is called once after all the rows in the aggregate have been processed and returns thecomputed value.
In Listing 7.12 we create a new aggregating function, median(), to calculate the median of a set of numbers. Themedian is a value such that half the values in the set are above the median and half are below it. The SQLite built-inavg() function computes the mean averagethe sum of all values divided by the number of values.
Listing 7.12. Creating an Aggregating Function Using the Perl Interface
my $sql = "SELECT median(num), AVG(num) FROM numbers";
my $sth = $dbh->prepare($sql)
or die("Cannot prepare: " . DBI::errstr() );
$sth->execute() or die("Cannot execute: ". DBI::errstr() );
my ($median, $mean) = $sth->fetchrow_array();
print "Median: $median \n";
print "Mean: $mean \n";
Let's look at the three methods that make up the median package, beginning with new().
sub new {
bless [];
}
This simple function is all that's needed to initialize an aggregator.
The step() method is also quite simple. Each $value that is to be aggregated is read in turn and pushed onto context.We do all the hard work in the finalize() function, so step() is just used to gather the required data into a workableformat.
sub step {
my ( $context, $value ) = @_;
push @$context, $value;
}
The finalize() method begins by reading context and making sure it contains at least one element.
my $context = $_[0];
if (!@$context) {
return undef;
}
If there are no elements from which to find the median, undef is returnedtranslated to a NULL result in SQL.
To find the median value we need to look at the elements in numerical order, so a sort is performed on context.
@sorted = sort { $a <=> $b } @context;
The rule to find the median where there is an odd number of values is simply to find the value of the middle element inthe list.
$count = @sorted;
if ($count%2 == 1) { # Odd number of elements
$median = $sorted[int($count/2)];
}
For an even number of values, the median is the midpoint between the two middle values.
Finally we return $median, which will become the result of the SQL aggregating function.
return $median;
We will use the data added by Listing 7.10 to test this new function, so we need some data to work from. This scriptcreated a table named numbers and inserted a sequence of square numbers into the num column.
The square numbers that were inserted, in order, are
1 4 9 16 25 36 49 64 81
We can see from looking at this sequence that the median will be 25, as there are four numbers on either side. Themean average, on the other hand, will be a bigger value as the later numbers are relatively large numbers compared tothe earlier numbers in the sequence.
Listing 7.12 fetches both the median and the mean average of the rows using the new user-defined function and theSQLite built-in:
$sql = "SELECT median(num), AVG(num) FROM numbers";
Running the script shows the calculated values as follows:
$ ./listing7.12.plMedian: 25
Mean: 31.6666666666667
Chapter 8. The Tcl Interface Tcl (Tool Command Language) is a flexible, portable scripting language interpreter that is ideal for rapid applicationprototyping. With its simple syntax and in conjunction with Tk, a graphical user interface toolkit shipped as standardwith Tcl, Tcl makes it possible to create powerful GUI applications very quickly.
In this chapter we look at how to add database functionality to your application using SQLite's Tcl interface.
Preparing to Use the Tcl Interface The Tcl interface is included with the SQLite source distribution. Compiling from source on a Unix system creates thelibrary file libtclsqlite.so, which should be installed to a valid library location.
A binary distribution of the TCL library can be obtained from http://www.sqlite.org/download.html for both Linuxand Windows, in compressed formats named tclsqlite.so.gz and tclsqlite.zip respectively.
The library file should go in a location from which Tcl can find the package. On Linux this would usually be asubdirectory of /usr/share/tcl; on Windows, sqlite.dll is typically extracted to C:\tcl\lib.
Assuming the subdirectory is called TclSqlite, running the following command from Tcl will generatepkgIndex.tclwhich is required in order for Tcl to be able to locate the packageon a Linux system:
If you installed using the RPM distribution on a compatible Linux system, tclsqlite.so will already have been installedto a directory from which it can be imported into tclsh or wish.
Using the Tcl Interface Now that you have the SQLite Tcl library installed, let's look at how the interface is used within a Tcl script. In thissection you'll see how to open and close a database and to issue commands that insert, update, or delete rows, andquery the database. You'll also learn how to handle the resulting dataset in your script.
Opening and Closing a Database
To begin using the SQLite library from Tcl, your scripts first have to import the package.
package require sqlite
Run interactively, tclsh will display the version number of the SQLite library.
$ tclsh% package require sqlite2.0
You can access SQLite via Tcl by using a single command named sqlite, which you call as follows:
sqlite dbcmd database-name
The database file given in the database-name argument is openedor created if it does not exist. SQLite will attempt toopen database-name from the current directory unless a path is specified.
A new composite command with the name given for dbcmd is then created for use within Tcl. The new commandinterfaces with an open SQLite database using a number of methods, similar to the way in which Tk widgets arecreated.
The first method we'll look at is the simplestthe close method instructs SQLite to close an open database, after whichit will destroy the dbcmd command.
The first Tcl command in the following example would open a database called tcldb from the current directory andbind it to the command db1 within Tcl. The second command then closes the database, after which the command db1will no longer be available.
$ tclsh% package require sqlite2.0
% sqlite db1 tcldb0x95c7eb0
% db1 close
As usual, the error messages Tcl displays are helpful and generally no further error-trapping code is required. Forinstance, if you had tried to execute the preceding code in a directory that you did not have permission to write to, theerror displayed would look like this:
$ tclsh% package require sqlite2.0
% sqlite db1 tcldbunable to open database: tcldb
Executing SQL Statements
SQL statements are passed to SQLite from Tcl via the eval method. In its simplest form, the syntax is as follows:
% dbcmd eval sql-statement
The script in Listing 8.1 shows how a simple table called contacts can be created from a Tcl script.
Listing 8.1. Executing a CREATE TABLE Statement Using the TCL Interface
package require sqlite
sqlite db1 tcldb
db1 eval {
create table contacts (
id INTEGER PRIMARY KEY,
first_name CHAR,
last_name CHAR,
email CHAR UNIQUE
)
}
The SQL command in Listing 8.1 is formatted using spaces and newlines to indent and separate the columns withinthe CREATE TABLE statement. There is no need to format the SQL statements passed to the eval method in anyparticular way; however, this formatting is stored in the sqlite_master database and will be retrieved exactly as it wastyped when the schema is subsequently queried.
Executing the script should be silent unless there is an error, for instance if the tcldb database cannot be written to orthe table cannot be created. If the script in Listing 8.1 is run a second time, for instance, the following error messagewill be displayed:
As usual, the error message produced is verbose enough that no further error trapping is required to find out whatwent wrong.
Using Commands That Change the Database
The script in Listing 8.2 prompts for three pieces of user input before generating an SQL statement to insert a newrecord into the contacts table.
Listing 8.2. Inserting a New Record with the TCL Interface
package require sqlite
puts "Enter first name:"
gets stdin first_name
puts "Enter last name:"
gets stdin last_name
puts "Enter email:"
gets stdin email
sqlite db1 tcldb
set qry "INSERT INTO CONTACTS (first_name, last_name, email)
VALUES ('$first_name', '$last_name', '$email')"
db1 eval $qry
In this listing, we have taken an extra step to assign the SQL query to a new variable, qry, which is then passed to theeval database method. The following output shows successful execution of this script.
This program terminates silently if there are no errors, or will return a fairly readable error message if there is aproblem. As the email column has the UNIQUE attribute, we can see how an error from an INSERT statement isreported by trying to insert another record with the same email address.
The first line of the output contains the error column email is not unique and tells us the command that caused the errorand the line it appeared on. The SQL command itself is not shown on screen and you would need to output thecontents of qry to the screen to see it verbatim; although, in this case it's fairly obvious what the query would havebeen.
Another reason this script might fail is if there are single quote characters in the user input. When evaluating the query,the strings read from stdin are replaced literally, so if we try to add a name containing an apostrophe (the same ASCIIcharacter as the single quote in SQL), it causes a syntax error as shown in the following example:
The error message this time is not so helpful, but if we consider the query that has actually been passed to the evalmethod we can see where the problem arises.
INSERT INTO CONTACTS (first_name, last_name, email)
The apostrophe has skewed the pairing of single quotes in the querynow 'O' and ', ' are contained in quotes, and theSQL statement is nonsense. The first error the SQLite parser comes across is the @ sign occurring outside of quotes,so that is the error reported.
To avoid this problem you should escape single quotes before they are used inside an SQL statement. This can bedone easily using string map. The following example shows how to modify the code from Listing 8.2 to grab user inputfor the last_name field to eliminate any problems with apostrophes in the name:
puts "Enter last name:"
gets stdin last_name
set last_name [string map {' ''} $last_name]
The string map command finds any occurrence of one single quote in last_name and replaces it with two quotes. Theresulting string is assigned back to last_name. SQLite uses two adjacent single quotes to represent a single quotewithin a string, so with this change in place the SQL generated by our script for the previous example now looks likethis:
INSERT INTO CONTACTS (first_name, last_name, email)
For this example only the last_name field was a problem, but you should consider all of your user input to ensure thatsuch problems are avoided.
The INSERT statement we've used in the preceding example creates an implicit transaction in which the single newrecord is added to the database. Where an explicit transaction is required around a group of statements, you caninvoke a BEGIN TRANSACTION command through the SQLite Tcl interface, and similarly end the transaction witha COMMIT TRANSACTION or ROLLBACK TRANSACTION instruction.
Listing 8.3 shows how several INSERT commands can be executed in a single transaction using SQLite's Tclinterface.
Listing 8.3. Using a SQLite Transaction from Tcl to Insert Multiple Rows
package require sqlite
sqlite db1 tcldb
db1 eval { BEGIN TRANSACTION }
db1 eval {
INSERT INTO CONTACTS (first_name, last_name, email)
In Listing 8.3 we begin by passing the BEGIN TRANSACTION command to SQLite and end the transaction bysaving changes to the database with the COMMIT TRANSACTION command.
At the very end of the script, we use a new method on the db1 command. The last_insert_rowid method returns themost recently assigned value of the INTEGER PRIMARY KEY fieldin this example, the id field on the contacts table.Running the script will produce output similar to the following:
$ tclsh listing8.3.tclLast rowid inserted: 5
When an UPDATE or DELETE operation is performed on a database, a number of rows will be changed orremovedassuming of course that the WHERE clause finds some matching rows that the operation is to be performedupon. To find how many rows were affected, we can use the changes database method.
Listing 8.4 performs an UPDATE on the contacts table that converts the first_name and last_name fields touppercase using the SQL function upper(). As there is no WHERE clause, every row in the table will be affected andthe value returned by the changes method will be the total number of rows in the table.
Listing 8.4. Using the changes Method to Find How Many Rows Were Affected by an UPDATE
package require sqlite
sqlite db1 tcldb
db1 eval {
UPDATE CONTACTS
SET first_name = upper(first_name),
last_name = upper(last_name)
}
puts "Rows updated: [db1 changes]"
Running this script will result in the changes being made to the database and the number of rows that have beenchanged will be displayed.
$ tclsh listing8.4.tclRows updated: 5
Note
The number of affected rows returned by the changes method is the total number of rows that could be affected bythe operation regardless of whether or not the record is changed by itin other words the number of rows matched bythe WHERE clause or the total number of rows in the table if no WHERE clause is given. If you were to run the scriptin Listing 8.4 a second time, it will still show the same number of changes having been made, even though the stringsare already in uppercase and their values are not changed.
Fetching Records from the Database
We have already seen how to pass an SQL statement to SQLite using the eval database method. Now let's look athow the result of a SELECT statement can be processed in Tcl.
To simply assign the entire result set from a SELECT statement to a variable in Tcl, you can assign the return value ofthe eval method as shown in Listing 8.5.
Listing 8.5. Assigning the Output of a SELECT Statement to a Variable
package require sqlite
sqlite db1 tcldb
set result [db1 eval { SELECT first_name, last_name FROM contacts } ]
puts $result
When the script in Listing 8.5 is executed, the output will be similar to the following:
$ tclsh listing8.5.tclCHRIS NEWMAN PADDY O'BRIEN TOM THOMAS BILL WILLIAMS JO JONES
This could be more useful. Although everything we asked for in the SELECT has indeed been assigned to result, thereis nothing to indicate what is what. To the eye it's fairly clear that each pair of words output makes up a full name, butthe data is not in a format that we can work with in a program. This method of assigning the result of a query is reallyonly suitable for simple, single-row results.
Fortunately the Tcl interface allows you to process the data returned by a SELECT query one row at a time byspecifying the name of an array variable and a script immediately after the SQL code. The value of each column willbe inserted into the named array and the supplied code will be executed once for each row in the query result.
Listing 8.6 shows how the same query on the contacts table can be processed one row at a time.
Listing 8.6. Processing the Result of a SELECT Query One Row at a Time
package require sqlite
sqlite db1 tcldb
db1 eval {
SELECT first_name, last_name FROM contacts
} result {
parray result
}
The code to be executed for each row returned is simply to dump the contents of the result array to screen usingparray. By doing so we can see exactly what elements are created in this array, as shown in the following output:
Not only are the selected values made available as elements of result using their column names as the keys, but theirrespective data types are also fetched into an element with a key named typeof:column_name.
Additionally, for each row returned, result(*) will contain a list of columns returned by the query. If your query wasSELECT * FROM tablename, the value of result(*) is, usefully, the list of actual column names returned and not justthe * character.
If the name of the array variable given is an empty string, the value of each column will be stored in a scalar variablewith the same name as the column itself and the subsequent code executed only once. This property of the SQLite Tclinterface is demonstrated in Listing 8.7.
Listing 8.7. Processing the Result of a SELECT Query One Row at a Time
package require sqlite
sqlite db1 tcldb
db1 eval { SELECT * FROM contacts } {} {
puts "$first_name $last_name"
}
This time the values of all the columns in the contacts table are fetched into variables with the same names as thecolumns, so we can output the values from the first_name and last_name columns by referencing the first_name andlast_name variables. The following output shows the result of running Listing 8.7:
$ tclsh listing8.7.tclCHRIS NEWMAN
PADDY O'BRIEN
TOM THOMAS
BILL WILLIAMS
JO JONES
The onecolumn Database Method
If you want to select only a single value from the database, the onecolumn database method can be used in place ofeval. The behavior is identical except that only the first column of the first row of the result is returned, regardless ofthe number of columns and rows in the result of the query.
This is a convenience function only; the same behavior can be achieved with the eval method and using lindex in Tclto fetch the first column of the first row. The following two pieces of code behave in the same way:
set result [db1 onecolumn { SELECT * FROM contacts }]
set result [lindex [db1 eval { SELECT * FROM contacts }] 0]
Validating an SQL Statement
The SQLite Tcl interface provides a database method that allows you to check whether or not an SQL statement iscomplete before attempting to execute it using eval or onecolumn. This method is called complete and is called asfollows:
if [ dbcmd complete sql-statement ] { # Action if statement is complete
} else {
# Action if statement is not complete
}
Note
The complete method is not a check for valid SQL syntax. Its underlying function is sqlite_complete() from theC/C++ library, for which the logic is little more than to make sure an SQL statement ends with a semicolon. It is alittle cleverer than thisfor instance, semicolons can appear within a CREATE TRIGGER commandbut the statement isnot complete until the semicolon following the keyword END.
The script in Listing 8.8 reads one line of input at a time and continues until a complete SQL statement has beenentered.
Listing 8.8. Checking Whether an SQL Statement Is Complete
package require sqlite
sqlite db1 tcldb
gets stdin qry
while { ![ db1 complete $qry ] } {
gets stdin line
set qry "$qry $line"
}
puts "Query is complete:"
puts $qry
A typical execution of this script might look like this:
$ tclsh listing8.8.tclSELECT *FROM contactsWHERE email like '%.co.uk';Query is complete:
SELECT * FROM contacts WHERE email like '%.co.uk';
By default, SQLite will not wait for a lock to clear on the database file before giving up on a database transaction.SQLite can support multiple simultaneous processes reading from a database, but everything must stop when a writeaction takes placeno reading or other writing can be done at the same time.
There will be situations where you want to adjust the length of time that SQLite will wait for a lock to clear beforegiving up. This can be done with the timeout method, called like this:
dbcmd timeout ms
The value ms is the maximum length of time to wait, in milliseconds. Therefore a value of 1000 would mean themaximum delay is one second.
The busy method is similar to the timeout method, but offers you more control over what action should be takenwhen a lock cannot be obtained within the specified amount of time.
The busy method specifies a Tcl procedure to be used as a callback whenever a locked database cannot be opened.The callback can perform any action you want before returning to the main program to try to obtain a lock again. Thecallback procedure takes a single count parameter, which tracks how many times it has been called, and must returnzero to force the calling script to retry the database transaction. A non-zero value will cause the operation to beinterrupted.
Listing 8.9 shows how the busy callback can be used. This example simply dumps off to the callback if the databaseis locked and prints a message to the screen. The script will allow the callback to be called five times before giving upon the lock.
Listing 8.9. Using a Busy Callback to Handle a Locked Database
package require sqlite
sqlite db1 tcldb
proc mycallback {file count} {
puts "This is the callback function, try $count"
if {$count < 5} {
# Do something useful whilst waiting
return 0
} else {
return 1
}
}
db1 busy mycallback
db1 eval { UPDATE contacts SET first_name = upper(first_name) }
Unless the tcldb database is already locked, this script will execute silently. To test the busy callback, lock thedatabase for writing by issuing a BEGIN TRANSACTION command from another session. The following output isproduced when the script fails to obtain a lock on five successive tries:
$ tclsh listing8.9.tclThis is the callback function, try 1
This is the callback function, try 2
This is the callback function, try 3
This is the callback function, try 4
This is the callback function, try 5
database is locked
while executing
"db1 eval { UPDATE contacts SET first_name = upper(first_name) }"
(file "listing8.9.tcl" line 17)
Adding New SQL Functions
The SQLite Tcl extension allows the SQL language to be extended with new functions implemented as Tcl code.This is a very powerful feature that means you can extend SQL on demand using a familiar language, without having tolearn C or recompile SQLite.
Note
Although the Tcl extension supports regular user-defined functions, aggregating functionswhich are supported bysome other language interfacescannot be created in Tcl.
The function database method is used to specify a Tcl function that is to be registered as an SQL function. It takestwo arguments, the name of the Tcl function itself and the name with which it is to be referenced in SQLthe names canbe different.
Listing 8.10 shows how a user-defined function that reverses the characters in a string can be added to SQL.
Listing 8.10. Registering a User-Defined SQL Function Using the Tcl Library
The Tcl function in this example is called reverse_string and takes a single string argument. The first thing we do is findthe length of this string to set up a loop.
set len [string length $str]
Then we perform a loop taking one character at a time from the end of the string and building a new string in reverseorder. Because the first character of a string is found at index 0, the loop actually runs from len-1 to 0.
for {set x [expr $len - 1]} {$x>=0} {incr x -1} {
set rev $rev[string index $str $x]
}
Finally, with the reversed string stored in rev, we return the result of the operation.
return $rev
We register this function in SQL using the function method. Listing 8.10 assigns a different name to the SQL functionthan the Tcl functionthe SQL function is to be called simply reverse.
db1 function reverse reverse_string
Finally, to test our custom function we perform a SELECT on a fixed string. Of course, now that the SQL function isavailable, it can be applied to a database column or expression.
Executing the sample script gives the following output, as expected:
dlrow ,olleH
Chapter 9. The Python Interface Python is an interpreted object-oriented programming language that is portable across different platforms. Itspopularity is due to a combination of its powerful features and the clarity and agility of its syntax. It is an ideal languagefor prototyping while still managing to keep the code readable and manageable.
You can access SQLite from Python by using the PySQLite module, available from http://pysqlite.sourceforge.net/.PySQLite provides an interface to SQLite compliant with the Python Database API Specification 2.0, so theprogramming interface should be familiar if you have used Python with another database in the past and should still beintuitive if not.
The Python Database API Specification can be found at http://www.python.org/peps/pep-0249.html.
Preparing to Use the Python Interface Before you can install PySQLite, you need to be sure your system has the Python interpreter and the developmentlibraries installed, as well as a C compiler. You also need to have a SQLite development librarylibsqlite.so orsqlite.dllinstalled, as PySQLite does not include it.
Some Linux distributions package the Python interpreter and development libraries separately, and simply selectingPython at install time may not have included the development package. On Red Hat Linux, for example, you need tomake sure the python-devel package has been installed.
Download the PySQLite source from http://sourceforge.net/projects/pysqlite/ and extract it. The version number inthe filename may be different from that shown.
$ gunzip -c pysqlite-0.5.1.tar.gz | tar xf -
Enter the pysqlite directory that has been created and issue the following command to compile PySQLite:
The following step installs the PySQLite extension into system directories and so you will need to become the rootuser, if you are not already, using the su command.
# python setup.py installrunning install
running build
running build_py
running build_ext
running install_lib
[...]
To test that PySQLite has installed correctly, run the test suite that is shipped with the source code. The followingoutput indicates a successful installation:
Using the Python Interface Now that you have the PySQLite interface installed, let's look at how the interface is used from a Python script. Inthis section we'll see how to open and close databases and issue commands to SQLite.
Opening and Closing a Database
In order to use the PySQLite functionality, Python must first import the sqlite module. Then you can call thesqlite.connect() constructor method in order to open a database. The following is an example with Python running ininteractive mode:
$ pythonPython 2.3.3 (#1, May 7 2004, 10:31:40)
[GCC 3.3.3 20040412 (Red Hat Linux 3.3.3-7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
In its simplest form, the constructor takes a single database name parameter. The filename given can include a relativeor absolute path and will be read from the current working directory if no path is specified. If that file exists, thatdatabase is opened; otherwise, a new empty database is created with the given filename. A connection object isreturned, assigned to cx in this example.
If you followed the preceding example and no error occurred, exit Python and you'll see that the file pydb has beencreated in the current directory.
To close the connection to a database, simply call the .close() method on the connection object.
>>> cx.close()
After the connection has been closed, cx becomes unusable and an Error exception will be raised if you attempt toperform any operation on it.
Executing SQL Commands
To perform any database operation you must first create a cursor object using a valid database connection object.SQLite does not use the concept of cursors internally; however, PySQLite emulates this behavior in its interface inorder to provide compliance with the Python Database API Specification. This is done using the .cursor() method, asfollows:
>>> cx = sqlite.connect("pydb")>>> cu = cx.cursor()
Note
You can actually create as many cursor objects as you like on the same database connection, although there isusually little point in doing so. Separate cursors do not isolate datachanges made in one cursor are immediatelyreflected in other cursors created from the same connection.
With a cursor object, you can use the .execute() method to execute an SQL command. Listing 9.1 shows a simplePython script that opens the pydb database and issues a CREATE TABLE statement to create the contacts table.
Listing 9.1. Executing a CREATE TABLE Statement Using the Python Interface
import sqlite
sql = """
CREATE TABLE contacts (
id INTEGER PRIMARY KEY,
first_name CHAR,
last_name CHAR,
email CHAR UNIQUE
)
"""
cx = sqlite.connect("pydb")
cu = cx.cursor()
cu.execute(sql)
cx.commit()
cx.close()
Run this script to create the table, and verify that it has been created using the sqlite tool.
$ python listing9.1.py$ sqlite pydbSQLite version 2.8.15
Enter ".help" for instructions
sqlite> .tables
contacts
sqlite> .schema
CREATE TABLE contacts (
id INTEGER PRIMARY KEY,
first_name CHAR,
last_name CHAR,
email CHAR UNIQUE
);
Error Handling
If for any reason the database file cannot be opened or the query cannot be executed, Python will raise an error. Thefollowing output was generated when an attempt was made to create a new SQLite database in a write-protecteddirectory:
File "/usr/lib/python2.3/site-packages/sqlite/__init__.py", line 61,
in connect
return Connection(*args, **kwargs)
File "/usr/lib/python2.3/site-packages/sqlite/main.py", line 445,
in __init__
self.db = _sqlite.connect(database, mode)
_sqlite.DatabaseError: unable to open database: pydb
This is pretty ugly, so you'll probably want to trap any such errors and handle them in a nice way. The StandardErrorerror class can be used to trap an error from PySQLite, but the sqlite package also includes its own Error class, withsubclasses for more specific error detection.
Listing 9.2 performs the same CREATE TABLE statement as before, but uses the sqlite.Error exception so that theerror is handled within the script.
Listing 9.2. Using sqlite.Error to Trap Errors
import sys;
import sqlite;
sql = """
CREATE TABLE contacts (
id INTEGER PRIMARY KEY,
first_name CHAR,
last_name CHAR,
email CHAR UNIQUE
)
"""
try:
cx = sqlite.connect("pydb")
except sqlite.Error, errmsg:
print "Could not open the database file: " + str(errmsg)
sys.exit()
try:
cu = cx.cursor()
cu.execute(sql)
cx.commit()
except sqlite.Error, errmsg:
print "Could not execute the query: " + str(errmsg)
sys.exit()
cx.close()
When this script is run from a location in which it does not have permission to open the database file, the output willlook like the following:
$ ./listing9.2.pyCould not open the database file: unable to open database: pydb
If file permissions are okay but you run the script after the table has already been created, this will be the errorproduced:
$ ./listing9.2.pyCould not execute the query: table contacts already exists
PySQLite provides the subclasses of Error specified by the Database API specification in order to handle specifictypes of error message. The different error classes are shown in Table 9.1.
Table 9.1. Error Subclasses Implemented by PySQLite
InterfaceError Error returned by the database interface.
DatabaseError Error returned by the database engine when a databasecannot be opened, or when SQLite returnsSQLITE_ERROR, SQLITE_READONLY,SQLITE_CORRUPT, SQLITE_FULL,SQLITE_CANTOPEN, SQLITE_SCHEMA, or anunknown error code.
DataError Indicates a problem with data values when SQLitereturns SQLITE_TOOBIG, for example division byzero.
OperationalError Error related to the operation of the database that is outof the developer's control, when SQLite returnsSQLITE_PERM, SQLITE_ABORT, SQLITE_BUSY,SQLITE_LOCKED, SQLITE_INTERRUPT,SQLITE_IOERR, or SQLITE_PROTOCOL.
IntegrityError Error due to a problem with the referential integrity of thedatabase, when SQLite has returnedSQLITE_CONSTRAINT or SQLITE_MISMATCH.
InternalError An internal error in the database engine, raised whenSQLite returns SQLITE_INTERNAL,SQLITE_NOTFOUND, or SQLITE_EMPTY.
ProgrammingError Indicates a programming error, caused by an error in auser-defined function or an operation being attempted ona closed database.
NotSupportedError Raised if a non-supported operation is attempted.
Connection Parameters
Several optional arguments can be supplied to sqlite.connect() after the name of the database file. The full prototypeis
The mode argument is currently not used but was intended to allow you to supply a file mode in which the databasewill be opened. Mode 0644 could be used, for instance, to open a database read-only.
The converters argument can be used to supply a mapping from SQL to Python data types. We will discuss theissues surrounding Python and SQLite data types in the section "Working with Data Types."
The autocommit feature is off by default but can be turned on for a connection object by setting this argument to 1.We will discuss this feature later in this chapter in the "Transactions" section.
Use the encoding argument to tell PySQLite what encoding method you want to use to save Unicode strings to thedatabase. For example, pass the value utf-8 as the encoding argument to use UTF-8 encoding.
The timeout argument takes a value in seconds, which is the amount of time you want SQLite to continue trying toobtain a lock on the database file before giving up with a DatabaseError exception. The default value is None,meaning that no special action will be taken, and only one attempt will be made to open the database.
You can use the command_logfile argument to specify a file object that all the raw SQL statements passed to theSQLite engine through PySQLite will be logged to.
Using Commands That Change the Database
The script in Listing 9.3 prompts for three pieces of user input before generating an SQL statement to insert a newrecord into the contacts table.
Listing 9.3. Inserting a Database Record Using Data from User Input
import sys
import sqlite
try:
cx = sqlite.connect("pydb")
except sqlite.Error, errmsg:
print "Could not open the database file: " + str(errmsg)
sys.exit()
fn = raw_input("Enter first name : ")
ln = raw_input("Enter last name : ")
em = raw_input("Enter email address: ")
sql = """
INSERT INTO contacts (first_name, last_name, email)
VALUES ('%s', '%s', '%s')
""" % (fn, ln, em)
try:
cu = cx.cursor()
cu.execute(sql)
cx.commit()
except sqlite.Error, errmsg:
print "Could not execute the query: " + str(errmsg)
sys.exit()
cx.close()
The three calls to raw_input() gather information from standard input that we'll use to build the SQL INSERTstatement. The first name, last name, and email are stored to fn, ln, and em respectively.
fn = raw_input("Enter first name : ")
ln = raw_input("Enter last name : ")
em = raw_input("Enter email address: ")
We then use these values to build up a string, sql, which contains the command to be passed to SQLite.
sql = """
INSERT INTO contacts (first_name, last_name, email)
VALUES ('%s', '%s', '%s')
""" % (fn, ln, em)
The record is then inserted when we open a cursor on the SQLite connection and execute the SQL statement. Thecommit() method is called to ensure that the record is saved before closing the database connection.
Executing the script to add a record to the contacts table looks like this:
$ python listing9.3.pyEnter first name : ChrisEnter last name : NewmanEnter email address: [email protected]
We can then verify that the record has been inserted using the sqlite tool.
$ sqlite pydbsqlite> SELECT * FROM contacts;id first_name last_name email
Care must be taken to ensure that records read from user input are safe for insertion into SQLite. The followingexample shows how a name containing an apostrophe character can break the script in Listing 9.3:
Enter first name : PaddyEnter last name : O'BrienEnter email address: [email protected] not execute the query: unrecognized token: "@"
If you were to output the value of sql before it was executed, it would look like this:
INSERT INTO contacts (first_name, last_name, email)
SQLite is unable to determine that the apostrophe in O'Brien is not the closing quote around the second data value. Itconsiders only values within pairs of single quotes to be the data items.
In this case, the values to be inserted will be seen as 'Paddy', 'O' and ', ', and so the SQL statement is nonsense. It isthe @ symbol that first causes a tokenizer error and the script to raise a DatabaseError exception.
Quotes are delimited in SQLite by another quote characteryou have to use two single quotes wherever you want oneto appear. The preceding SQL statement would work if it were written as follows:
INSERT INTO contacts (first_name, last_name, email)
However, to save having to validate user input and delimit apostrophes yourself, the execute() method can be used tosubstitute values into the query in a safe way. Rather than substituting values into sql, consider the following changes.
If we alter the SQL statement so that values are not enclosed in quotes at all, we can make PySQLite do the hardwork of determining whether values require quoting and, if necessary, how special characters need delimiting.
sql = """
INSERT INTO contacts (first_name, last_name, email)
VALUES (%s, %s, %s)
"""
Then we execute the statement by passing a list of substitution values as the second argument to .execute().
cu.execute(sql, (fn, ln, em) )
The modified version can be seen in full in Listing 9.4.
Listing 9.4. Using .execute() to Substitute Values into a Query
import sys
import sqlite
try:
cx = sqlite.connect("pydb")
except sqlite.Error, errmsg:
print "Could not open the database file: " + str(errmsg)
sys.exit()
fn = raw_input("Enter first name : ")
ln = raw_input("Enter last name : ")
em = raw_input("Enter email address: ")
sql = """
INSERT INTO contacts (first_name, last_name, email)
VALUES (%s, %s, %s)
"""
try:
cu = cx.cursor()
cu.execute(sql, (fn, ln, em) )
cx.commit()
except sqlite.Error, errmsg:
print "Could not execute the query: " + str(errmsg)
sys.exit()
cx.close()
We can now insert records using strings that contain apostrophes into SQLite without having to worry aboutdelimiting quotes first.
$ python listing9.4.pyEnter first name : PaddyEnter last name : O'BrienEnter email address: [email protected]$ sqlite pydbsqlite> SELECT * FROM contacts;id first_name last_name email
When a database operation is performed that affects rows in the databasenamely UPDATE and DELETEoperationsyou can find the number of rows affected by the command by using the .rowcount attribute of the cursorobject.
In Listing 9.5 we perform an UPDATE command that changes the case of the first_name field in contacts touppercase across all database rows. To show how many rows were actually affected, the .rowcount attribute isdisplayed.
Listing 9.5. Finding the Number of Rows Affected by an SQL Statement
import sys
import sqlite
try:
cx = sqlite.connect("pydb")
except sqlite.Error, errmsg:
print "Could not open the database file: " + str(errmsg)
sys.exit()
sql = "UPDATE contacts SET first_name = upper(first_name)"
try:
cu = cx.cursor()
cu.execute(sql)
cx.commit()
except sqlite.Error, errmsg:
print "Could not execute the query: " + str(errmsg)
sys.exit()
print str(cu.rowcount) + " row(s) were affected"
cx.close()
When you execute this script, every row in the database is updated and the number of rows is printed to screen.
$ python listing9.5.py2 row(s) were affected
Note
The number of affected rows returned by .rowcount is the total number of rows that the UPDATE or DELETEstatement considersin other words, the number of rows matching the WHERE clause, if one is given. If a row isupdated to the same value it held before the statement was executed, it is still counted by .rowcount even though nochange to the data has been made.
After a SELECT statement, .rowcount also contains the number of rows returned by the query. In fact it returns thenumber of rows returned or affected by the WHERE clause in any SQL statement.
Transactions
PySQLite contains some built-in optimization to make the best use possible of SQLite's transactions. When a numberof INSERT, UPDATE, or DELETE statements are issued in succession in a script, they are all executed within asingle transaction.
The default behavior, if the SQL commands were entered through sqlite or via the underlying C/C++ interface, wouldbe to open an implicit transaction for each statement and commit each transaction before moving on to the nextcommand. There is significant performance benefit to be gained from grouping multiple statements into a singletransaction, and so PySQLite attempts to handle the grouping automatically.
Whenever the .execute() method is passed an SQL statement that changes the database, PySQLite automaticallysends a BEGIN TRANSACTION before that command is executed if a transaction has not already been started. A.commit() therefore needs to be issued before the changes will be saved to the database.
Note
If you look back to Listing 9.1, you'll see that we included a .commit() instruction before closing the database. Notethat calling .close() while a transaction is open will roll back any unsaved changes.
PySQLite opens a transaction whenever necessary but is optimized to leave it as late as possible before the BEGINTRANSACTION command is sent so that the database file is locked for as little time as possible. It is therefore thetype of statement processed by .execute() that determines whether a transaction is started.
SELECT statements do not need to be run inside a transaction but if the result of a query is subsequently used inanother statement that does change the database, you should make sure that both SQL commands are executedwithin the same transaction.
This is the default behavior of PySQLite, although it can be changed if desired by turning on autocommit mode byusing a value of 1 for the autocommit argument in sqlite.connect().
cx = sqlite.connect("pydb", 755, {}, 1)
Because autocommit is the fourth argument, the mode and converters arguments must be supplied. In this example wehave simply given them their default values.
An alternative syntax that is more compact is given in the following example. PySQLite recognizes that the secondparameter is not a file mode and applies the autocommit setting:
cx = sqlite.connect("pydb", autocommit=1);
When autocommit is turned on, PySQLite will not attempt to do anything clever with transactions. An INSERT,UPDATE, or DELETE command issued outside of a transaction will be executed within its own implicit transaction. Ifyou want to start a transaction manually, you must pass BEGIN TRANSACTION to the .execute() method.
Fetching Records from the Database
Let's take a look at how the result of a SELECT query is processed using PySQLite. The query is submitted through.execute() just as other types of SQL statement are executed.
To fetch the data retrieved by the SELECT, call .fetchone() on the cursor object. Data is returned one row at a timeeach time .fetchone() is called, and returns None if there are no more rows to be fetched.
Listing 9.6 shows a simple example that selects first_name and last_name from the contacts table. Each data recordis fetched into row, which is then used to output the two column values to screen.
Listing 9.6. Fetching Selected Data Using .fetchone()
import sys
import sqlite
try:
cx = sqlite.connect("pydb")
except sqlite.Error, errmsg:
print "Could not open the database file: " + str(errmsg)
sys.exit()
sql = "SELECT first_name, last_name FROM contacts"
try:
cu = cx.cursor()
cu.execute(sql)
except sqlite.Error, errmsg:
print "Could not execute the query: " + str(errmsg)
sys.exit()
row = cu.fetchone()
while row:
print row.first_name + " " + row.last_name
row = cu.fetchone()
cx.close()
The while loop simply prints the contents of row and, assuming there are more records to come, fetches the next rowand repeats. The names of the selected columns are also the names of the elements of rowwe referencerow.first_name and row.last_name respectively.
To comply with the Python Database API Specification, fetched columns can also be referenced using the indexnumber of their position in the list, starting at zero. Take this example:
print row.first_name + " " + row.last_name
This could also be written as
print row[0] + " " + row[1]
The following is an example of output from running Listing 9.6:
$ python listing9.6.pyCHRIS Newman
PADDY O'Brien
Bill Williams
Complimentary to .fetchone(), PySQLite also implements the .fetchmany() and .fetchall() methods to retrieve themultiple rows or even the entire data set returned by the SELECT all at once.
The .fetchmany() method takes a single argument to specify the number of rows to fetch. If the value is smaller thanthe number of rows remaining, the cursor position is advanced so that the next call to .fetchone() or .fetchmany() willbegin at the next row in sequence.
The .fetchall() method, as you have probably guessed, is equivalent to calling .fetchmany() with the argument equal tothe total number of rows returned by the query. If .fetchone() or .fetchmany() has been called on that cursor objectpreviously, only the remaining rows will be fetched.
Listing 9.7 performs the same query on the contacts table using .fetchmany() to grab the result two records at a time.
Listing 9.7. Fetching Several Rows at a Time Using .fetchmany()
import sys
import sqlite
try:
cx = sqlite.connect("pydb", 755)
except sqlite.Error, errmsg:
print "Could not open the database file: " + str(errmsg)
sys.exit()
sql = "SELECT first_name, last_name FROM contacts"
try:
cu = cx.cursor()
cu.execute(sql)
except sqlite.Error, errmsg:
print "Could not execute the query: " + str(errmsg)
This time, we use two nested loops in order to display all the records. The outer while loop fetches rows two at atime, and the inner for loop iterates each of the fetched rows into a single row.
The .rowcount attribute for SELECT statements contains the total number of rows returned by the query. We outputthis value in Listing 9.7 to show the number of records that will be fetched.
$ python listing9.7.pyQuery returns 3 row(s)
CHRIS Newman
PADDY O'Brien
Bill Williams
If no value is given, the number of rows is taken from the .arraysize attribute, which will contain the optimum numberof rows to fetch at a time.
For example, the following alternative could have been used in Listing 9.7:
cu.arraysize = 2
rows = cu.fetchmany()
Working with Data Types
As we saw in Chapter 2, "Working with Data," SQLite is typeless. However, Python uses strong data typing andalthough most of the time it is able to infer what type it needs to convert a string from SQLite to, sometimes you needto give a mapping to ensure that a data value is converted to the correct type in Python.
Although the column types given to SQLite in the CREATE TABLE statement are mostly ignoredother than todetermine whether numeric or text-based sorting is performed on that columnPySQLite will look at the column typeto try to determine how a column should be converted to a Python data type.
Table 9.2 shows the default mappings, which should be kept in mind when creating the schema for a database that isto be interfaced from PySQLite.
Table 9.2. Default Mappings of SQLite Columns to Python Data Types
Column Type Converter Function
CHAR, TEXT str()
INT int()
FLOAT, NUMERIC, NUMBER, DECIMAL, REAL,and DOUBLE
float()
UNICODE UnicodeConvertor(self.encoding)
BINARY, BLOB sqlite.decode()
DATE DateTime.DateFrom()
TIME DateTime.TimeFrom()
TIMESTAMP DateTime.DateTimeFrom()
INTERVAL DateTime.DateTimeDeltaFrom()
Note
The column types shown in Table 9.2 can be substrings of the actual type specified in the table schema. For example,an INTEGER column will also be converted using int(). Column types are not case-sensitive. The closest match foundis used and sometimes you must be careful; for example, INT is closer to INTEGER than INTERVAL. To be sure ofusing the correct column type, always give the name in full.
To override the default mapping converter functions, the converter name can be passed to .execute() before a queryis executed, in the following format:
cu.execute("-- types type1, type2, ...")
In this example, col1, col2, and so on are the converter functions to use for each column in turn. For example, to tellSQLite to convert the first two columns returned from the next query executed as string and integer respectively, youwould use the following command:
cu.execute("-- types str, int")
The following built-in types can be passed in a -- types command:
str
int
long
float
unicode
binary
date
time
timestamp
interval
Adding New SQL Functions
Though not part of the Python Database API Specification, PySQLite provides an interface to SQLite's powerfuluser-defined function capabilities for both simple and aggregating functions.
This feature allows you to create your own functions in Python that can be registered as SQLite functions andtherefore called within an SQL statement. There is therefore no need to know C in order to extend the SQL languagewhen using PySQLite.
User-defined functions are registered through the .create_function() method on a database connection object.Functions are available in SQL only for the duration of that connectionthey are not saved to the database itself ormade available to other connection objects within the same script.
In Listing 9.8 we create a function, altcaps(), in Python and then register it as an SQL function. The function takes asingle string argument and returns it with alternating capitalization. This function is fairly useless but should serve as anexample of how user-defined functions are implemented via PySQLite.
Listing 9.8. Creating a User-Defined SQL Function
import sys
import string
import sqlite
def altcaps (word):
altstr = ""
for i in range(len(word)):
if i%2:
altstr += string.lower(word[i])
else:
altstr += string.upper(word[i])
return altstr
try:
cx = sqlite.connect("pydb")
except sqlite.Error, errmsg:
print "Could not open the database file: " + str(errmsg)
sys.exit()
cx.create_function("altcaps", 1, altcaps)
sql = "SELECT altcaps(first_name) FROM contacts"
try:
cu = cx.cursor()
cu.execute('-- types str')
cu.execute(sql)
except sqlite.Error, errmsg:
print "Could not execute the query: " + str(errmsg)
sys.exit()
row = cu.fetchone()
while row:
print row[0]
row = cu.fetchone()
cx.close()
The altcaps() function uses range() to create a loop that will run once for each character in word.
for i in range(len(word)):
We then use the mod operator (%) to test whether the loop index is odd or even. A new string altstr is built up acharacter at a time, with every odd-numbered character added in uppercase and every even position in lowercase.
if i%2:
altstr += string.lower(word[i])
else:
altstr += string.upper(word[i])
This function is then registered in SQLite:
cx.create_function("altcaps", 1, altcaps)
There are three arguments to .create_function()the name of the SQL function, the number of arguments, and areference to the Python function that is to be executed when the SQL function is called.
It is important to tell PySQLite what data type to expect to be returned from a user-defined function; otherwise, floatwill be assumed. We set a string type for this function with the following line :
cu.execute('-- types str')
Note
PySQLite assumes that user-defined functions return a NUMERIC unless told otherwise. Therefore the defaultconverter function is float().
Executing Listing 9.8 produces the following output, consisting of the first_name values from contacts with alternatingupper- and lowercase letters:
$ python listing9.8.pyChRiS
PaDdY
BiLl
In this example we gave the SQL function the same name as the Python function, although this is not a requirement.In Listing 9.9 we reference Python's random() function from the standard library and create an SQL function calledpyrandom() so as not to conflict with SQLite's built-in random() function.
Note
If you use .create_function() to create an SQL function with the same name as an existing function, the user-definedfunction will simply overload the existing function. No error or warning is raised.
Listing 9.9. Registering a Python Library Function in SQL
import sys
import random
import sqlite
try:
cx = sqlite.connect("pydb")
except sqlite.Error, errmsg:
print "Could not open the database file: " + str(errmsg)
sys.exit()
cx.create_function("pyrandom", 0, random.random)
sql = "SELECT pyrandom()";
try:
cu = cx.cursor()
cu.execute(sql)
except sqlite.Error, errmsg:
print "Could not execute the query: " + str(errmsg)
sys.exit()
row = cu.fetchone()
print row[0]
cx.close()
Adding Aggregating Functions
PySQLite also allows you to add custom aggregating functions to SQL. An aggregating function is one that applies acalculation across many rows at a timeeither the entire dataset returned by a query or rows grouped together by aGROUP BY clause.
Aggregating functions require a class to be defined containing four functions:
A step function to be executed for each row, named step()
A function to be executed after each row has been processed, named finalize()
A function to reset data stored in the class, named reset() when a new aggregate calculation begins
We will create a class definition that can be used to register an aggregating function to compute the median of a set ofnumbers. First, we need some data to try the function on.
Listing 9.10 creates a simple table, numbers, and loops from 1 to 9 inserting the square of each value. Because thesequence of square numbers does not increase proportionally, the mean and median averages will be different.
Listing 9.10. Creating a Table Containing a Sequence of Square Numbers
import sys
import sqlite
try:
cx = sqlite.connect("pydb")
except sqlite.Error, errmsg:
print "Could not open the database file: " + str(errmsg)
sys.exit()
cu = cx.cursor()
sql = "CREATE TABLE numbers (num INTEGER)"
cu.execute(sql)
for i in range(1,10):
sql = "INSERT INTO numbers VALUES (%d)" % (i*i)
cu.execute(sql)
cx.commit()
cx.close()
The sequence inserted looks like this:
1 4 9 16 25 36 49 64 81
The median is a value such that half the values in the set are above the median and half are below it. From looking atthe sequence we can see that the median will be 25.
Listing 9.11 contains the class definition for our median() aggregating function and executes an SQL statement to findboth the median and the meanusing the SQLite built-in AVG() functionon the sequence of square numbers.
Listing 9.11. Creating a Table Containing the Mean and Median Averages
print "Could not open the database file: " + str(errmsg)
sys.exit()
cx.create_aggregate("median", 1, median)
sql = "SELECT median(num), avg(num) FROM numbers";
try:
cu = cx.cursor()
cu.execute(sql)
except sqlite.Error, errmsg:
print "Could not execute the query: " + str(errmsg)
sys.exit()
row = cu.fetchone()
print "Median is " + str(row[0])
print "Mean is " + str(row[1])
cx.close()
The class contains the following member functions:
def __init__(self):
self.reset()
The constructor calls reset() to initialize the local array in which the values from which the median is calculated will bestored.
def reset(self):
self.values = []
The step() function simply appends each value to the local array. Nothing more is required here; the hard work isdone in finalize().
def step(self, val):
self.values.append(float(val))
First of all we sort the array into numerical order.
def finalize(self):
self.values.sort()
The rule to find the median is slightly different depending on whether there is an odd or even number of elements.
If there are an odd number of elements, the median is simply the middle value from the list. The index of the midpointof the array is half the length of the array, rounded down to the nearest whole number.
n = len(self.values)
if n%2 == 1:
return self.values[n//2]
If there is an even number of elements, the median is the midpoint between the two middle values.
The aggregating function is registered in SQLite using .create_aggregate().
cx.create_aggregate("median", 1, median)
The syntax is similar to .create_function() except that the argument is the name of the class that contains the reset(),step(), and finalize() methods required to implement that aggregate.
Listing 9.11 executes the following query to find both the median and mean averages from the test data:
SELECT median(num), avg(num) FROM numbers
The following output is produced to show that the median and mean are indeed different values:
$ python listing9.11.pyMedian is 25.0
Mean is 31.6666666667
Part III: SQLite Administration10 General Database Administration
Chapter 10. General Database Administration This chapter deals with SQLite administration topics. We will look at how the database parameters can be adjustedwith the PRAGMA command and discuss ways to back up and restore your database. We will also examine some ofthe database internals that will help you to gain a better understanding of the SQLite engine.
The PRAGMA Command The PRAGMA command provides an interface to modify the operation of the SQLite library and perform low-leveloperations to retrieve information about the connected database.
Fetching Database Information
This section describes the set of PRAGMA directives that allows you to find information about the currently attacheddatabases.
PRAGMA database_list;
One row is returned for each open database containing the path to the database file and the name that the databasewas attached with. The first two rows returned will be the main database and the location in which temporary tablesare stored.
A PRAGMA command can be executed from the sqlite program as well as through any of the language APIsafter all,the sqlite program is just a simple front end that passes typed commands to the C/C++ interface and uses a choice ofcallback function, selected using .mode, to output the rows to screen.
The following example shows the result of executing PRAGMA database_list through the sqlite program after asecond database has been attached to the session:
sqlite> ATTACH DATABASE db2 AS db2;sqlite> PRAGMA database_list;seq name file
0 main /home/chris/sqlite/admin/db1
1 temp /var/tmp/sqlite_6x5rZ2drAEtVpzj
2 db2 /home/chris/sqlite/admin/db2
If executed from a language interface, the pseudo-table returned by this pragma contains columns named seq, name,and file.
To find information about a database table, use the table_info pragma with the following syntax:
PRAGMA table_info(table-name);
The table-name argument is required and can refer only to a table in the main database, not one attached usingATTACH DATABASE. One row is returned for each column in that table, containing the columns shown in Table10.1.
Table 10.1. Columns in the Pseudo-Table Returned by PRAGMA table_info
cid An integer column ID, beginning at zero, that shows theorder in which columns appear in the table.
name The name of the column. The capitalization used in theCREATE TABLE statement is retained.
type The data type of the column, taken verbatim from theCREATE TABLE statement.
notnull If the column was declared as NOT NULL, this columnwill contain a non-zero value; otherwise, it will be zero.
dflt_value Contains the DEFAULT value for the column if one hasbeen specified, otherwise NULL.
pk Will be 1 for the column(s) making up the PRIMARYKEY on the table and 0 for all other columns.
To find information about the indexes on a table, use the index_list and index_info pragmas as follows:
PRAGMA index_list(table-name);
PRAGMA index_info(index-name);
PRAGMA index_list causes one row to be returned for each index on the given table-name with the columns shownin Table 10.2.
Table 10.2. Columns in the Pseudo-Table Returned by PRAGMA index_list
seq An integer beginning at zero that indicates the order inwhich the indexes were added to the table.
name The name of the index. This is the identifier given in aCREATE INDEX statement, or the assigned name if theindex was created with a PRIMARY KEY or UNIQUEconstraint in the CREATE TABLE statement.
unique Will be 1 if the index was declared as UNIQUE;otherwise, it will be 0.
When you know the name of an index, PRAGMA index_info can be used to find out which column or columns makeup that index. A single column index will return a single row, whereas a clustered index will return one row for eachindexed column, in the relevant order. The columns returned are shown in Table 10.3.
Table 10.3. Columns in the Pseudo-Table Returned by PRAGMA index_info
seqno An integer beginning at zero that indicates the sequenceof the columns in the index
cid The cid number of the indexed column, from table_info
name The name of the indexed column
When an index is created implicitly by the use of a UNIQUE or PRIMARY KEY constraint in the CREATE TABLEstatement, the index will take an automatically assigned name of the form (table-name autoindex num). The followingexample shows these pragmas being run for a table with both explicit and implicit index names:
sqlite> PRAGMA index_list(mytable);seq name unique
---- ------------------------------ ------
0 col4_idx 0
1 (mytable autoindex 2) 1
2 (mytable autoindex 1) 1
sqlite> PRAGMA index_info('(mytable autoindex 2)');seqno cid name
----- ---- ----------
0 0 col1
1 1 col2
Note
If the table-name or index-name argument is not found in the current database, no error is returned. The result of thePRAGMA is simply silent.
Altering Database Parameters
This section describes the set of PRAGMA directives that enables you to adjust certain database parameters for thepurposes of performance tuning at the overall database level. The first parameter that can be adjusted is cache_size.
PRAGMA cache_size;
This instruction will return the maximum number of database disk pages that will be held in memory at once. Thedefault value of 2000 pages equates to around 3000KB of memoryeach page taking up approximately 1.5KB.
The same PRAGMA can be used to set a new cache size by assigning a value as follows:
PRAGMA cache_size = num-pages;
Increasing the cache size can improve performance if the SQL operations you are executing would use a largeramount of memory than the size of the cache. For instance, an UPDATE operation on a large number of rows wouldbenefit from a larger cache size. Keeping a limit on the cache size ensures that SQLite's memory usage remains underyour control.
The cache_size can be altered dynamically, so if one query in your application in particular can benefit from a largernumber of pages being cached, you can increase the size for that query and drop it back down afterwards.
The temp_store pragma allows the location used for the temporary database to be queried or changed.
PRAGMA temp_store;
PRAGMA temp_store = value;
The value of temp_store can be one of the constants DEFAULT, MEMORY, or FILE, which have the values 0, 1,and 2 respectively. The values indicate whether SQLite should store temporary database objects in memory or todisk. The DEFAULT value is set at compile time and unless you have changed it, it will be FILE.
Using in-memory databases for temporary tables can produce significant performance savings if your application canafford the additional memory usage.
The synchronous pragma allows the synchronous flag for the current database session to be queried or changed.
PRAGMA synchronous;
PRAGMA synchronous = value;
The value can be OFF, NORMAL, or FULL, which have the values 0, 1, and 2 respectively.
In synchronous mode FULL or NORMAL, SQLite will pause periodically to make sure data has been written to thedisk before carrying on with the next operation. FULL mode is very safe and ensures that after a system crash for anyreason, even unrelated to SQLite, the database will not be corrupt.
Because FULL synchronous is very slow, NORMAL mode is used more often. SQLite still pauses to check atcritical moments, but there is a small chance that an operating-system crash or power failure at the wrong time couldcause corruption of the database file.
When synchronous is OFF, SQLite does not pause to check that data has been written to disk. This mode is veryfastsome operations have been clocked at 50 times faster than NORMAL modehowever, the chance of data loss isgreater because it relies on the operating system to ensure that data has been written. Setting synchronous=OFF cangive some major performance benefits, but you should give careful consideration to the potential consequences.
A good time to turn synchronous OFF is when populating a large database for the first time. In this situation, the riskof corruption is not critical because the population process could be restarted from a known point in the event of afailure.
Note
NORMAL mode is used by default, even though there is a small chance of data loss, for performance reasons. Inreality, the kind of system crash that would cause your database to become corrupt is likely to carry with it other,more serious problems such as hardware failure, so NORMAL mode is considered acceptable for everyday use.
Altering Database Parameters Permanently
Changes made in the way shown in the preceding section will persist only until the current database session is closed.To change a database parameter permanently, the same PRAGMA directives can be called with a default_ prefix toquery and set the value that is stored in the database file.
To query the current default cache size of the attached database, use the following instruction:
PRAGMA default_cache_size;
To increase the cache size for that database to 4000 pages permanently, the command would be
PRAGMA default_cache_size = 4000;
The same prefix can be applied to give default_temp_store and default_synchronous pragmas to query or set thepermanent values of temp_store and synchronous respectively.
Altering Query Parameters
Though it can now be done using the sqlite_changes() function call, the traditional method of counting changes madeto the database as a result of an INSERT, UPDATE, or DELETE operation was to use the count_changesPRAGMA. This PRAGMA is likely to be removed from future versions of SQLite in favor of sqlite_changes(), so itis included here only for historical value.
PRAGMA count_changes = ON;
PRAGMA count_changes = OFF;
When count_changes is set to ON, a single row is returned whenever an operation is performed that modifies thedatabase containing the number of affected rows. The following example shows this in action using the sqlite program:
sqlite> DELETE FROM contacts WHERE first_name = 'CHRIS';1
Unless you are working with legacy code that uses count_changes, there is a small performance gain to be had byleaving this pragma turned off.
Note
The examples in this chapter use ON or OFF as the Boolean argument where one is required. ON, trUE, and YESare equivalent to a value of 1, whereas OFF, FALSE, and NO are equivalent to zero.
The column names passed to the callback function in the C/C++ interface as columnNamesalso used in otherlanguage APIs to identify the name of the column returnedare usually just the names of the columns, with a table prefixused only where a join has taken place. Use the full_column_names pragma to force SQLite to report a column nameas table.column, even when only one table has been queried.
PRAGMA full_column_names = ON;
The following output shows an example of this in the sqlite monitor programthe column headings now include the tablename:
The columnNames parameter to the callback function will always contain the names of the columns selected inelements numbered from zero to argc1, but this array can be extended to also include the data types specified in theCREATE TABLE statement.
PRAGMA show_datatypes = ON;
Where no data type was specified in the CREATE TABLE statement, it is reported as NUMERIC. Expressionsevaluated in a query are reported as either NUMERIC or TEXT depending on the expression itself.
The example in Listing 10.1 shows how the show_datatypes pragma can be used to view the data types of bothdatabase columns and evaluated expressions.
Listing 10.1. Using PRAGMA show_datatypes to Find the Data Types of Columns and Expressions
#include <stdio.h>
#include <sqlite.h>
int callback(void *pArg, int argc, char **argv, char **columnNames)
{
int i;
for (i=0; i<argc; i++) {
printf("%-20s %-8s %s\n",
columnNames[i], columnNames[i+argc], argv[i]);
}
return(0);
}
int main()
{
char *errmsg;
int ret;
sqlite *db = sqlite_open("db1", 0777, &errmsg);
if (db == 0)
{
fprintf(stderr, "Could not open database: %s\n", errmsg);
The resulting output shows that the column types of the database columns are as they were declared in the CREATETABLE statement, but the expressions revert to one of the internal data types, NUMERIC or TEXT.
$ ./listing10.1id INTEGER 1
first_name CHAR Chris
id+2 NUMERIC 3
upper(first_name) TEXT CHRIS
Analyzing the Database
An integrity check can be performed on the entire database with the integrity_check pragma. It is simply issued asfollows:
PRAGMA integrity_check;
The check searches for corrupt indexes, malformed records, and other such problems. Any issues are returned as asingle string or if there are no problems, a single string ok is returned. The response you would hope to see is
sqlite> PRAGMA integrity_check;ok
The parser_trace pragma turns ON or OFF internal tracing of the SQLite parser stack and, when ON, returns onerow for each step of the parser trace. To use this feature, SQLite must have been compiled without the NDEBUGcompile time option.
The following example shows the first few lines of output from the parser trace for a simple SELECT statement. Asyou can see, it is not desirable to have this output turned on unless you are looking for a particular problem with theparser codethere were 108 lines of output in total, just for this simple query!
sqlite> PRAGMA parser_trace=ON;sqlite> SELECT first_name, last_name FROM contacts;parser: Input SELECT
parser: Reduce [explain ::=].
parser: Shift 3
parser: Stack: explain
parser: Shift 73
parser: Stack: explain SELECT
parser: Input ID
parser: Reduce [distinct ::=].
parser: Shift 74
parser: Stack: explain SELECT distinct
parser: Reduce [sclp ::=].
parser: Shift 289
parser: Stack: explain SELECT distinct sclp
parser: Shift 64
parser: Stack: explain SELECT distinct sclp ID
parser: Input COMMA
parser: Reduce [expr ::= ID].
parser: Shift 290
[...]
Similarly, tracing can be enabled at the Virtual Database Engine level using the vdbe_trace pragma. One row isreturned for each VDBE opcode in the trace, which gives a result very similar to that obtained by the EXPLAINcommand.
sqlite> PRAGMA vdbe_trace=ON;sqlite> DELETE FROM contacts; 0 Transaction 0 0
1 VerifyCookie 0 3
2 Transaction 1 0
3 Clear 3 0
4 SetCounts 0 0
5 Commit 0 0
6 Halt 0 0
We will examine the VDBE opcodes in more detail later in this chapter.
Backing Up and Restoring Data It is, of course, essential to back up your data regularly. SQLite stores databases to the filesystem, so backing upyour databases can be as easy as taking a copy of the database files themselves and putting them somewhere safe.
However, if you are copying the database file, can you be sure that it is not being written to at the time you issue thecopy command? If there's even a small chance that a SQLite write operation could happen during the copy, youshould not use this method to back up your database to ensure that data corruption cannot occur.
SQLite implements locking at the database level using a lock on the database file itself, which works to the extent thatother processes using the SQLite library know when the database is being written to. Many processes can read froma database at the same time; however, a single writer process locks the entire database, preventing any other read orwrite operation taking place.
On Windows platforms, if SQLite has locked the file the operating system will acknowledge the lock and will notallow you to copy the file until SQLite is done with it. Likewise the copy operation will lock the database file so thatSQLite cannot access it. However, on Unix systems, file locks are advisoryin other words they tell you that the file islocked if you care to check, but do not otherwise prevent access. Therefore it is entirely possible to initiate a cpcommand while SQLite has a database file locked for writing.
The .dump Command
The .dump command in sqlite provides a simple way to create a backup file in the form of a list of SQL statementsthat can be used to re-create the database from scratch. The CREATE TABLE statements are preserved and oneINSERT command is written for each row of data in the table.
Note
Because the .dump command is part of an application that uses the SQLite library, there are no issues with filelocking to worry about. If the database is locked at the moment you attempt to fetch the rows from the database, thebusy handler will be invoked, which, in sqlite, is to wait for the specified timeout before returning an error message.
A backup operation can therefore be performed as follows:
$ sqlite dbname .dump > backup.sql
Future restoration from the generated backup.sql is simple:
$ sqlite dbname < backup.sql
You can optionally pass a table name to .dump to extract only the schema and records for that table. The followingexample shows the dump file produced for the contacts table:
Note that the INSERT statements generated do not contain a column list before the VALUES keyword. If you areusing the .dump output to restore a table exactly as it used to be, this is no problem; however, you cannot alter thetable schema in the CREATE TABLE statement without also modifying every INSERT in the extract file.
The .dump command can act as a handy workaround for the lack of ALTER TABLE command in SQLite if you areable to adjust the INSERT commands accordingly. The following example uses sed to change the SQL commands tofull inserts so that you can go ahead and add columns to the CREATE TABLE statement to re-create the databasefrom this file with the new schema.
$ echo .dump | sqlite dbname | \ sed e 's/VALUES/(id,first_name,last_name,email) VALUES/g' \ > backup.sql
Whether you decide to use .dump for regular backups or to simply copy the database file itself will depend on severalfactors. The output from .dump is often smaller than the SQLite database file and will certainly compress much smallerusing gzip or some similar utility.
However, restoring from an SQL file can take some time, particularly if there are a lot of indexes to be rebuilt afterthe data has been loaded. As a backup should be as quick and easy to restore as possible, copying the SQLitedatabase file may be the correct choice.
Using .dump is a better choice for a long-term archive of old data. Although there is, in theory, no reason that thesame version of SQLite will not be available several years from now, you might actually want to load your archiveddata into a newer version of SQLite or even some other database system. The SQL output file produced by .dumpwill be fairly portable.
Exploring the SQLite Virtual Database Engine In this section we'll take a look under the hood of SQLite to see how the command process works and to examinethe internals of the Virtual Database Engine (VDBE). Though it is not essential to know what goes on behind thescenes, a little insight into how the SQLite engine processes your queries can help you understand SQLite better, andif you want to gain a deeper knowledge of how SQLite is implemented, you will learn the roles of the different sourcecode files.
SQLite Architecture
Let's begin by looking at the steps SQLite goes through in order to process a query. Figure 10.1 shows a diagram ofthe flow of information between submitting a query for processing and the output being returned.
Figure 10.1. Architecture of the SQLite Database Engine
Interface
The interface is the C library through which all SQLite processing is programmed. Even if you are using a differentlanguage API, at a lower level it is implemented using the C library. The core interface is defined in the main.c file ofthe SQLite source distribution. A few other files contain ancillary functions; for instance, printf.c containssqlite_mprintf() and its derivatives.
After a command is received through the interface, it is passed into the SQL Command Processor, which consists ofthree separate steps.
Tokenizer
The tokenizer handles the process of breaking up a character string containing SQL statements into tokens that canbe understood by the parser. Each individual component of the SQL language is a tokeneach combination ofcharacters that forms a keyword or identifier and each symbol or sequence of symbols that forms an operator.
Parser
SQLite uses the Lemon LALR(1) parser generator, which is probably not already on your system, so the SQLitesource code is distributed with the entire Lemon source code in a single file as tool/lemon.c. Though it does much thesame job as BISON or YACC, Lemon was chosen for its ability to generate a parser that is thread-safe and lessprone to memory leaks.
The source file parse.y contains the grammar definition for SQLite's implementation of SQL. Lemon reads this fileand generates a corresponding C-language file for the actual parser.
There is a document giving an introduction to the Lemon parser at http://www.hwaci.com/sw/lemon/lemon.html.
Code Generator
The parser assembles the tokenized SQL components into statements, checking that its syntax is valid before callingthe Code Generator to produce virtual machine code that will do the actual work of executing the SQL statement.We'll look at the way the virtual machine works shortly.
Many SQL statements have their own source code file to handle the conversion to virtual machine code. Forinstance, select.c, insert.c, update.c, and delete.c deal with SELECT, INSERT, UPDATE, and DELETE statementsrespectively. The WHERE clause is processed by where.c, and code to evaluate expressions is generated by expr.c.
Virtual Machine
The virtual machine implements an abstract computing engine for the specific purpose of manipulating database files.The virtual machine itself can be found in the source code files vdbe.c, and various utilities used to compile SQLstatements to VDBE opcodes are contained in vdbeaux.c.
Backend
SQLite uses an abstraction layer between the virtual machine and the low-level storage and retrieval routines,implemented as a B-tree, page cache, and OS interface.
B-Tree
The database is stored to disk using a B-tree layera balanced tree structure that provides fast data access byminimizing the number of disk reads required to access a particular piece of information. The implementation andinterface to SQLite's B-tree subsystem can be found in btree.c and btree.h.
Each table and index in the database is written using a separate B-tree though, as we have already seen, all theB-trees are stored in a single file on disk.
Pager
The B-tree storage mechanism requests data from the disk in 1KB chunks. The pager layer implements a pagecaching system that handles the reading, writing, and caching of these chunks, including the rollback and commitoperations needed to implement atomic transactions.
The pager implementation and its interface can be found in pager.c and pager.h respectively.
OS Interface
For portability between different platforms, SQLite has implemented an abstraction layer that interfaces with theunderlying operating system.
The file os.c contains routines that deal with low-level filesystem access on Windows and Unix. The definitions arecontained in preprocessor ifdef instructions to ensure that the relevant interface is built into SQLite at compile time.
Red/Black Tree
SQLite stores in-memory databases using a red/black balanced binary tree structure, defined in btree_rb.c. Nopager or OS interface is required for this part of the backend as the data is stored to RAM only.
Virtual Machine Opcodes
The VDBE's machine language contains more than 100 opcodes, defined in vdbe.c, that are used to handle everypossible database operation. We will look at how a few common operations are translated into opcodes and use thevdbe_trace pragma to walk through the command execution process.
Each instruction in the virtual machine is made up of an opcode and between zero and three operands. The firstoperand, named P1, must be an integer. P2 is also an integer but must be non-negative. P2 will always contain thejump destination for an operation that may cause a jump. P3 can be either a NULL-terminated string, or simplyNULL.
To see SQL commands converted to VDBE opcodes as they are executed from the sqlite program, begin by turningon the vdbe_trace pragma. To use this feature, SQLite must have been compiled without the NDEBUG compile timeoption.
sqlite> PRAGMA vdbe_trace=ON; 0 Halt 0 0
Immediately we see some VDBE trace output, although not a lot has happened for it to trace just yet. The zero at thestart of the line is the instruction numberas we'll see shortly, lines from the trace are numbered to indicate the sequenceof operationfollowed by the Halt opcode with operands P1 and P2 both zero.
The Halt opcode instructs the VDBE to exit immediately, tidying up any incomplete operations. We will always seethe Halt instruction at the end of every command trace.
P1 will contain the return code, which is usually SQLITE_OK, which is defined with a value of zero. The full list ofreturn codes can be found in Appendix E, "C/C++ Interface Reference."
When P1 is non-zero, the return code indicates an error has occurred and the value of P2 tells the virtual machinewhether to roll back the current transaction. The values OE_Fail, OE_Rollback, and OE_Abortdefined insqliteInt.hwill cause the transaction to be saved, rolled back, or aborted.
A different set of operands to Halt would be seen if we violate a database constraint, for instance. The followingoperation was the final trace step of a command that attempted to insert a duplicate value into a PRIMARY KEYcolumn:
14 Halt 19 2 column id is not unique
Here P1 has a value of 19, which represents a return code of SQLITE_CONSTRAINT and P2 has a value ofOE_Abort (2). Additionally, P3 contains a textual error message indicating the SQL error that caused this constraintto be violated.
Let's look at a very simple SELECT statement that evaluates an expression without using any database table. Thefollowing trace output shows the VDBE opcodes along with the values on the stack as the operation takes place:
sqlite> SELECT 2+3 AS result; 0 ColumnName 0 1 result
1 ColumnName 1 0 NUMERIC
2 Integer 2 0 2
Stack: si:2
3 Integer 3 0 3
Stack: si:3 si:2
4 Add 0 0
Stack: i:5
5 Callback 1 0
5
6 Halt 0 0
The first two opcodes on lines 0 and 1 are both ColumnName and contain the name and data type of the resultcolumn respectively. P1 is the number of the column, beginning at zero, for which the name in P3 is being assigned. P2has a value of 1 to indicate the last column in the result. Remember that, with the show_datatypes pragma on, the datatypes are passed to the callback as later column index numbers in the result set.
The Integer opcode causes the integer value of P1 to be pushed onto the stack. P3 contains a string representation ofthe same integer. After line 2 the stack trace shows si:2, where si indicates that the value was placed onto the stack asboth a string and an integer.
After line 3 the stack contains the values from both sides of the addition operator, and then the Add operation isperformed. Add does not take any operands; it pops the top two elements from the stack, adds them together, andpushes the result back onto the stack. After this operation we can see that indeed the stack contains the value 5;however, this time it is shown as i:5 as the Add opcode has created only an integer result.
With the hard work done, the Callback opcode is called. Callback pops P1 values off the stack and invokes thecallback function with an array of those values passed as the argv parameter.
The remainder of the trace shows the output of the callback function and a normal Halt operation.
Adding two integers isn't a complex operation, but it still has to be converted to VDBE opcodes in order to beevaluated by SQLite.
Let's look at an example of a database operation next. Listing 10.2 contains the virtual machine program codegenerated by the EXPLAIN command for a SELECT query that fetches two columns from the contacts table usingan equality condition in the WHERE clause.
Listing 10.2. Virtual Machine Program Generated by EXPLAIN
sqlite> .explain
sqlite> EXPLAIN ...> SELECT first_name, last_name ...> FROM contacts ...> WHERE first_name = 'Chris';addr opcode p1 p2 p3
Although EXPLAIN shows the complete program code that would be executed for a given SQL statement, thestatement is not executed. However, the vdbe_trace pragma produces its response while the query is being executed.The trace shows the commands as they are executed, and any jumps that alter the flow of the program result incorresponding jumps in the trace output.
Listing 10.2 shows the VDBE trace for the same query featured in Listing 10.1. The operation numbers are the samein both listings so that you can compare the lines in the trace to the complete virtual machine code.
sqlite> PRAGMA vdbe_trace=on;sqlite> SELECT first_name, last_name ...> FROM contacts ...> WHERE first_name = 'Chris'; 0 ColumnName 0 0 first_name
1 ColumnName 1 1 last_name
2 ColumnName 2 0 CHAR
3 ColumnName 3 0 CHAR
4 Integer 0 0
Stack: i:0
5 OpenRead 0 4 contacts
6 VerifyCookie 0 3455
7 Rewind 0 15
8 Column 0 1
Stack: s[Chris]
9 String 0 0 Chris
Stack: t[Chris] s[Chris]
10 StrNe 1 14
11 Column 0 1
Stack: s[Chris]
12 Column 0 2
Stack: s[Newman] s[Chris]
13 Callback 2 0
Chris|Newman
14 Next 0 8
8 Column 0 1
Stack: s[Paddy]
9 String 0 0 Chris
Stack: t[Chris] s[Paddy]
10 StrNe 1 14
14 Next 0 8
8 Column 0 1
Stack: s[Tom]
9 String 0 0 Chris
Stack: t[Chris] s[Tom]
10 StrNe 1 14
14 Next 0 8
8 Column 0 1
Stack: s[Bill]
9 String 0 0 Chris
Stack: t[Chris] s[Bill]
10 StrNe 1 14
14 Next 0 8
8 Column 0 1
Stack: s[Jo]
9 String 0 0 Chris
Stack: t[Chris] s[Jo]
10 StrNe 1 14
14 Next 0 8
15 Close 0 0
16 Halt 0 0
In operation 5, the OpenRead opcode opens a read-only cursor for a database table with the root page at the valueof P2 in the database identified by P1. In this example the contacts table is opened from root page 4 of the maindatabase. The temporary database has value 1 and any attached databases are given subsequent sequential numbers.
The VerifyCookie instruction checks that the schema version number of database P1 is equal to the value of P2. Theschema version number is an arbitrary integer that changes whenever the schema is updated and is used so thatSQLite knows when the schema has to be reread.
Next a Rewind takes place so that the following Column instruction will begin from the first record in the table. TheP2 operand to Rewind is the jump location should the operation failin this case execution will jump to line 15 toperform an immediate Close and Halt.
The Column opcode in instruction 8 causes the column indicated by P2 to be pushed onto the stack. In this case P2is 1, indicating the first_name column (with the first column, id, having index 0). We can see that the stack contains thestring Chris, the first_name of the first row in the database.
Instruction 9 pushes the argument in the WHERE clause onto the stack ready to be compared. The comparison isperformed with the StrNe opcode, which pops two elements from the stack and jumps to P2 if they are not equal.
In this example StrNe is used to implement an equality test by jumping past the conditional statement that returnsexecution to the top of the loop if the two strings are not equal. For this first record, the strings are equal, so operationcontinues to instruction 13 and pops the two fields in the result off the stack and passes them to the callback function.
The Next opcode at instruction 14 is where the jump from StrNe ends up if the strings are not equal. The cursor inP1 is advanced to the next data row and, as long as there are more records to fetch, execution jumps to theinstruction number in P2.
Program execution continues for the remaining rows in the contacts table, with first_name values of Paddy, Tom, Bill,and Jo not being equal to the specified string, so no further Callback operations are performed.
Finally the Next instruction has no more rows to fetch and the virtual machine program calls a Close on the opentable cursor and does a Halt with return code SQLITE_OK in instruction 16.
The complete list of opcodes for the SQLite VDBE is extensive and too large to cover in this chapter. Acomprehensive list is maintained as part of the online documentation at http://sqlite.org/opcode.html if you find youwant to analyze a particular SQL command in this level of detail.
Access to the Database File Because SQLite uses a local filesystem to store its databases, the SQL language does not implement the GRANTand REVOKE commands that are commonly found in client/server database systems. They simply do not make anysense in this context.
Therefore the only access issues that need to be addressed are those associated with the database file itself and aredone at the operating-system level.
File Permissions
For SQLite to be able to open a database, the user under which a process is running must have permission to accessthat file on the operating system.
Quite simply, to open a database for reading the process must be able to open the file in read-only mode, and toopen it for writing the process must have both read and write access to the file.
On a Unix system, if the process runs as the owner of the database file, the file mode can be 0400 or 0600 for readand write access respectively. Mode 0444 would enable all users to access the database file as read-only, whereasmode 0644 would additionally grant write access to the owner of the file, and mode 0666 would allow writing by allusers respectively. More information about file permissions can be found from man chmod.
Locking and Timeouts
Remember that although there is no limit to the number of concurrent reads allowed by SQLite, a single write processwill lock the database and prevent any further read or write operations until it is complete.
It is therefore important to handle the SQLITE_BUSY return code if more than one process might access yourdatabase at a time. The more instances of SQLite there are running in your application, the greater chance that one ofthem will encounter a timeout when trying to access the database file.
The simplest approach is to continue attempting the query in a loop until SQLITE_OK is returned or some other exitforces you to stop trying, with a small pause on each iteration of the loop. Listing 10.3 shows how you might takecare of this.
Listing 10.3. Handling the SQLITE_BUSY Status [View full width]
#include <stdio.h>
#include <sqlite.h>
int main()
{
char *errmsg;
int ret;
sqlite *db = sqlite_open("db1", 0777, &errmsg);
if (db == 0)
{
fprintf(stderr, "Could not open database: %s\n", errmsg);
sqlite_freemem(errmsg);
return(1);
}
do {
ret = sqlite_exec(db, "INSERT INTO mytable (col1, col2) VALUES (1, 2)", NULL, NULL,
&errmsg);
switch(ret) {
case SQLITE_OK: break;
case SQLITE_BUSY: fprintf(stderr, "Database locked, sleeping...\n");
sleep(1);
break;
default: fprintf(stderr, "Error: %s\n", errmsg);
sqlite_freemem(errmsg);
return(ret);
}
} while (ret != SQLITE_OK);
sqlite_close(db);
return(0);
}
The do ... while loop ensures that the query is executed at least once, and if the return code in ret is SQLITE_OK,the loop will not repeat a second time. However, if the SQLITE_BUSY error is encountered, the program outputs amessage and sleeps for one second before the loop restarts and the query is attempted again.
Note
The sleep() function on Unix takes as its argument a number of seconds to wait. However, on Windows it requires anumber of milliseconds. Listing 10.3 assumes a Unix platform so will only sleep for one thousandth of a second underWindows unless the value passed to sleep() is adjusted to 1000. However, it is much more useful if your program cando some other task than just hanging around while waiting to obtain a database lock.
Upon encountering any other return code, the program will halt execution displaying the contents of errmsg.
Similar functionality can be implemented using sqlite_busy_handler() to set a callback function that is invokedwhenever a lock cannot be obtained.
Listing 10.4 shows a busy handler callback that will cause the program to attempt to obtain a lock five times, sleepingfor one second between each try and giving up after the fifth attempt.
Listing 10.4. Using a Busy Handler Callback Function
#include <stdio.h>
#include <sqlite.h>
static int myhandler(void *NotUsed, const char *dbname, int numtries) {
fprintf(stderr, "Database locked on try %d, sleeping...\n", numtries);
if (numtries >= 5)
return(0);
else {
sleep(1);
return(1);
}
}
int main()
{
char *errmsg;
int ret;
sqlite *db = sqlite_open("db1", 0777, &errmsg);
if (db == 0)
{
fprintf(stderr, "Could not open database: %s\n", errmsg);
sqlite_freemem(errmsg);
return(1);
}
sqlite_busy_handler(db, myhandler, NULL);
ret = sqlite_exec(db, "INSERT INTO mytable (col1, col2) VALUES (1, 2)",
NULL, NULL, &errmsg);
if (ret != SQLITE_OK) {
fprintf(stderr, "Error: %s\n", errmsg);
sqlite_freemem(errmsg);
return(ret);
}
sqlite_close(db);
return(0);
}
The busy callback function is registered with the line
sqlite_busy_handler(db, myhandler, NULL);
The function pointer myhandler references the callback function declared at the top of the listing. The callback takesthree arguments. The first is the pointer passed as the third argument to sqlite_busy_handler(), which allows arbitrarydata to be passed to the callback. We have not used this facility in this example. Secondly, the name of the databasetable or index that SQLite was attempting to access is passed in. The third argument contains the number of attemptsthat have been made to obtain a lock, and Listing 10.4 restricts the number of attempts that may be made by lookingat this value.
if (numtries >= 5)
return(0);
else {
sleep(1);
return(1);
}
If numtries exceeds 5, the sleep() instruction will not occur and a value of 1 is returned. When the busy callbackreturns zero, it instructs the query to continue attempting execution. A non-zero return value causes the query to beinterrupted and the SQLITE_BUSY error to be raised.
The following output shows what happens when the program in Listing 10.4 is run on a locked database:
$ ./listing10.4Database locked on try 1, sleeping...
Database locked on try 2, sleeping...
Database locked on try 3, sleeping...
Database locked on try 4, sleeping...
Database locked on try 5, sleeping...
Error: database is locked
Multithreaded Database Access
SQLite includes support for multithreaded database connections so that you can access your database from two ormore threads simultaneously.
The ThrEADSAFE preprocessor macro is used to determine whether SQLite is built in thread-safe mode. Enablethis feature by adding DTHREADSAFE=1 to an appropriate place in the Makefile. Precompiled binary distributionsare thread-safe by default for Windows systems, but not for Unix systems.
The most important thing to remember is that each thread must make its own call to sqlite_open() to obtain an sqlite*type pointer. The same sqlite* pointer should not be accessed from more than one thread at a time as unpredictableresults may be seen. The child process from the result of a fork() command should also open its own copy of thedatabase after being spawned.
Be aware that because SQLite reads the database schema only once upon opening the database, if one threadchanges the schema the other thread will not be able to see the amended schema until you close and reopen thedatabase connectionjust the same as would happen in a multi-process environment.
Part IV: AppendixesA Downloading and Installing SQLite
B Reference for the sqlite Tool
C SQL Syntax Reference
D PHP Interface Reference
E C Interface Reference
F Perl Interface Reference
G Tcl Interface Reference
H Python Interface Reference
I The Future of SQLite
Appendix A. Downloading and Installing SQLite This appendix will show you where to go online to download the latest version of SQLite, how to pick the correctdistribution, andif you decide on a source code installationhow to compile it.
Obtaining SQLite The download page for SQLite is http://www.sqlite.org/download.html and is split into three sections:
Precompiled Binaries for Linux
Precompiled Binaries for Windows
Source Code
Unless you have a particular requirement to build SQLite from source, a binary distribution will work fine for Linuxand Windows systems.
Version numbers shown in this appendix are current at the time of writing; however, you should check for the lateststable version.
RPM Installation for Linux
The easiest way to install SQLite on compatible Linux systems is to use the RPM packages. The rpm command isfound on the RedHat and Fedora Core Linux distributions, though other distributions may also support it.
There are two RPM packages available for SQLite, found in the precompiled binaries section of the download page.
sqlite-2.8.15-1.i386.rpm contains the shared library required to run dynamically linked SQLite applications, and thesqlite program.
sqlite-devel-2.8.15-1.i386.rpm contains the static library, header files, and documentation in the form of man pages.
You should obtain both packages and install using the following steps as the root user:
The sqlite package must be installed before sqlite-devel to satisfy the package dependencies.
Binary Installation for Linux
The non-RPM binary distribution of SQLite is split into three files.
sqlite-2.8.15.bin.gz is a statically linked version of the sqlite program. This can be used alone to access and createSQLite databases from the command line.
sqlite-2.8.15.so.gz is the shared library that is needed to compile programs using the C/C++ interface.
tclsqlite-2.8.15.so.gz is a library file containing the TCL bindings for SQLite as well as the C/C++ interface.
Installation of each package is done using the same basic stepsuncompress the file, move it to a suitable location onyour system, and set the appropriate file permission.
The sqlite program should reside in a location that is in your path, such as /usr/bin or /usr/local/bin and should beexecutable by all users. Run the following commands as the root user:
The libsqlite.so library should reside in a location where it will be found by the C compiler, such as /usr/lib or/usr/local/lib, and should also be world-readable with the execute bit set.
On most systems, the file /etc/ld.so.conf contains a list of locations in which the linker will attempt to find a library file.In some cases it may be necessary to run the following command to ensure that libsqlite.so can be found:
# /sbin/ldconfig
The TCL-enabled library should be copied to a location that tclsh can import it from, such as a subdirectory of/usr/lib/tcl or /usr/share/tcl.
Binary Installation for Windows
The binary distribution for Windows consists of three Zip files.
sqlite-2_8_15.zip contains the sqlite.exe monitor program and library sqlite.dll.
sqlitedll-2_8_15.zip contains only the same sqlite.dll that is shipped with sqlite.exe. You do not need this package ifyou are also installing sqlite-2_8_15.zip.
tclsqlite-2_8_15.zip contains tclsqlite.dll, the library file with TCL bindings included.
Installation of each of these is a case of extracting the contents of the Zip file using WinZip or a similar program andmoving the individual components to an appropriate location on your system.
You should either put sqlite.exe in a folder that is in your command path or add its location to the path variable. Thefollowing command adds C:\sqlite to the command path:
C:\>path %PATH%;C:\sqlite
The library files should be placed in the usual place for DLLs on your system, for exampleC:\WINDOWS\SYSTEM32.
Installing from Source Code
To install SQLite on a platform for which there is no binary distribution, your only option is to build from source. Twodifferent source code archives are provided in .zip and .tar.gz format.
The following steps deal with extracting and compiling SQLite from the .tar.gz format on a Unix system. Obtain thefile sqlite-2.8.15.tar.gz and extract with the following command:
# gunzip -c sqlite-2.8.15.tar.gz | tar xf
A directory called sqlite will be created. Go into this directory and run the configure command.
# cd sqlite# ./configurechecking build system type... i686-pc-linux-gnu
checking host system type... i686-pc-linux-gnu
checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
[...]
configure: creating ./config.status
config.status: creating Makefile
config.status: creating sqlite.pc
The output from configure is many screens of text while the program works out your system settings. When it hasfinished, you can commence compilation by issuing the make command, and if everything appears to be in order,install SQLite with make install.
# make# make install
SQLite will then be installed to the appropriate places on your system.
Appendix B. Command Reference for the sqliteTool This appendix lists the commands that can be used within the sqlite monitor program.
Dot Commands The dot commands for sqlite can be used to change the output format, fetch information about the database, andmodify some settings. In this appendix, each dot command is shown first followed by an explanation.
Obtaining a List of Dot Commands
.help
Displays the full list of dot commands for the installed sqlite with brief descriptions.
Changing the Output Format
.mode list
The default output mode. Displays one line per record, with each column separated by a specific character or string.The default separator is the pipe character (|).
.separator string
Changes the separator for list mode output to string.
.mode lines
Causes sqlite to output each column in the result of a query to be displayed on a line by itself, with the value prefixedby the name of the column. Subsequent records are separated by a blank line.
.mode columns
Displays one line per record with data aligned in fixed-width columns.
.width width1 width2 ... widthN
Specifies the width in characters for each column in turn, where width1 is the leftmost column returned by the query.
A width setting of 0 (the default) will automatically size the column to whichever is largest of: the length of the columnheading, the length of the value in the first row of data, or 10 characters.
.headers on|off
Determines whether column headings are displayed in column output mode.
.mode insert table-name
Causes a list of full INSERT statements to be generated for the records returned by the query. The table-nameargument determines the name of the table for the INSERT statements.
.mode html
The output is generated as an XHTML table with one set of <TR> tags and each column as a <TD> element. If.headers on has been specified, <TH> tags are used for the column headings.
.nullvalue string
Specifies a value to display wherever a NULL value appears in the result of the query.
.echo on|off
When echo is set to on, the SQL command being executed is included as the first line of the output.
.explain on|off
Specifies a fixed-width column output mode suitable for the result of the EXPLAIN SQL command. Using .explainoff reverts the output settings to their former values.
.show
Displays the current settings for echo, explain, headers, mode, nullvalue, output, separator, and width.
Reading SQL Commands from a File
.read filename
Executes one or more SQL commands from filename in turn. A syntax error in the file will cause an error to bedisplayed but will not prevent execution of the remaining commands.
Sending Output to a File
.output filename
Specifies that the output from queries should be sent to filename. If filename already exists, it is overwritten with thenew output.
.output stdout
Specifies that output should be sent to standard outputusually the screen, but sqlite may also be used in a shell scriptwith output redirection.
Getting Information about the Database Schema
.databases
Lists the names of currently attached databases and the paths to the database files.
.tables pattern
Lists all tables in the primary attached database that contain pattern. If no pattern is given, all tables in the databaseare listed. Note that tables in databases attached with the ATTACH DATABASE command are not listed.
.schema table-name
Shows the CREATE TABLE statement that was used to create table-name. If no table-name is given, the entiredatabase schema is displayed.
.dump table-name
Generates a list of SQL commands that contain the CREATE TABLE and INSERT statements required to re-createthe table specified. If no table-name is given, the entire database is dumped.
.indices table-name
Lists the names of each index in turn on the given table. The table-name argument is required.
Changing the sqlite Prompts
.prompt main continue
Allows you to customize the prompts displayed when using sqlite. The two arguments replace the main prompt(usually sqlite>) and the continuation prompt (usually ...>) respectively. If only one argument is given, it is assumed tobe the main prompt.
Setting the Database Timeout
.timeout ms
Specifies the amount of time in milliseconds that sqlite will wait for a filesystem lock to clear when attempting to opena database file. If the time is exceeded, an error message is returned. The default value is zero, instructing sqlite toalways return an error immediately.
Exiting the sqlite Tool
.quit
.exit
Either of these commands closes the running sqlite program. The program will also exit if an end-of-file character isreceived.
Appendix C. SQL Syntax Reference This appendix summarizes the SQL language understood by SQLite. It provides the syntax for all supported SQLstatements.
Naming Conventions A standard identifier name must begin with a letter or an underscore character, and may contain any number ofalphanumeric characters or underscores. No other characters can be used. There is no enforced upper limit on thelength of an identifier name. Names can be as long as you like, but don't make them so long that you dread having totype them in full each time.
Square brackets or double quotes can be used to indicate a non-standard identifier name to the SQLite parser.Identifier names enclosed in this way can include characters other than the underscore, including spaces and evenother square brackets, and this also allows you to use SQL keywords as identifiers.
Note
It is not generally a good idea to use non-standard identifier names in your database; although, square brackets canstill be used around standard identifier names without consequence if you are familiar with this syntax from SQLServer or Microsoft Access.
Reserved Keywords
Tables C.1 through C.3 list the reserved keywords in SQLite. Table C.1 shows the fallback keywords, which can beused as identifiers without being delimited.
Table C.1. Fallback Keywords in SQLite
ABORT AFTER ASC ATTACH BEFORE
BEGIN DEFERRED CASCADE CLUSTER CONFLICT
COPY CROSS DATABASE DELIMITERS DESC
DETACH EACH END EXPLAIN FAIL
FOR FULL IGNORE IMMEDIATE INITIALLY
INNER INSTEAD KEY LEFT MATCH
NATURAL OF OFFSET OUTER PRAGMA
RAISE REPLACE RESTRICT RIGHT ROW
STATEMENT TEMP TEMPORARY TRIGGER VACUUM
VIEW
Table C.3. System Object Names in SQLite
_ROWID_ MAIN OID ROWID SQLITE_MASTER
SQLITE_TEMP_MASTER
Table C.2 shows the normal keywords, which can only be used as identifier names if they are contained in squarebrackets or double quotes.
Table C.2. Normal Keywords in SQLite
ALL AND AS BETWEEN BY
CASE CHECK COLLATE COMMIT CONSTRAINT
CREATE DEFAULT DEFERRABLE DELETE DISTINCT
DROP ELSE EXCEPT FOREIGN FROM
GLOB GROUP HAVING IN INDEX
INSERT INTERSECT INTO IS ISNULL
JOIN LIKE LIMIT NOT NOTNULL
NULL ON OR ORDER PRIMARY
REFERENCES ROLLBACK SELECT SET TABLE
THEN TRANSACTION UNION UNIQUE UPDATE
USING VALUES WHEN WHERE
Table C.3 shows the names of some system objects in SQLite that can only be used as an identifier name for adifferent type of object.
SQL Command Syntax This section details the SQL command syntax understood by SQLite. For clarity SQL keywords are shown inuppercase; however, SQLite is not case sensitive. Keywords and identifiers can be typed in uppercase, lowercase, ormixed case, and different capitalizations of the same string can be used interchangeably.
Creating and Dropping Database Objects
The CREATE object and DROP object statements are used to create and drop database objects.
CREATE TABLE
To create a new database table, use CREATE TABLE.
CREATE [TEMP | TEMPORARY] TABLE table-name (
column-def [, column-def]*
[, constraint]*
)
A column in the CREATE TABLE statement is defined as follows:
name [type] [[CONSTRAINT name] column-constraint]*
To drop a table, use DROP TABLE.
DROP TABLE [database-name.] table-name
Column Constraints
The optional column-constraint is composed of one or more of these keywords: NOT NULL, DEFAULT,UNIQUE, PRIMARY KEY, CHECK, and COLLATE.
NOT NULL [ conflict-clause ]
NOT NULL enforces that the column must always contain a value. An error will be raised on any attempt to insert aNULL value into the column.
DEFAULT value
DEFAULT defines a value that the column should take if no value is given when a row is inserted.
UNIQUE [ conflict-clause ]
UNIQUE creates a UNIQUE INDEX on the column, ensuring that the same value cannot be entered into this columnmore than once. There can be more than one UNIQUE INDEX on a table if required.
PRIMARY KEY [sort-order] [ conflict-clause ]
PRIMARY KEY creates a UNIQUE INDEX on the column designated as primary key for the table. Additionally ifthe column type is INTEGER, this column is used internally as the actual key of the table and the value is assignedautomatically by SQLite if it is not specified when a row is inserted. Only one PRIMARY KEY can be specified oneach table.
CHECK ( expr ) [ conflict-clause ]
At the present time, the CHECK clause is ignored; however, SQLite allows one to be specified in the syntax forpossible future use.
COLLATE collation-name
COLLATE specifies the text-collating function to be used when comparing values in this column and can be eitherTEXT or NUMERIC.
A UNIQUE, PRIMARY KEY, or CHECK constraint clause can also be specified after the column-def sectionusing the following syntax:
UNIQUE ( column-list ) [ conflict-clause ]
PRIMARY KEY ( column-list ) [ conflict-clause ]
CHECK ( expr ) [ conflict-clause ]
CREATE INDEX
To create an indexa sorting key on a database tableuse CREATE INDEX.
CREATE [UNIQUE] INDEX index-name
ON [database-name.] table-name ( column-name [, column-name]* )
[ ON CONFLICT conflict-algorithm ]
There is no limit to the number of indexes that can be added to a single table, nor on the number of columns in anyindex.
To drop an index, use DROP INDEX.
DROP INDEX [database-name.] index-name
CREATE VIEW
To create a viewa pseudo-table based on a query on one or more tablesuse CREATE VIEW.
CREATE [TEMP | TEMPORARY] VIEW [database-name.] view-name AS select-statement
You can SELECT from a view in the same way as a regular table, but you cannot perform a COPY, DELETE,INSERT, or UPDATE operation on a view.
To drop a view, use DROP VIEW.
DROP VIEW view-name
CREATE TRIGGER
To create a triggera procedure that executes automatically on a type of database eventon a table use CREATETRIGGER.
CREATE [TEMP | TEMPORARY] TRIGGER trigger-name [ BEFORE | AFTER ]
database-event ON [database-name.] table-name
trigger-action
To create a trigger on a view, use this syntax instead:
CREATE [TEMP | TEMPORARY] TRIGGER trigger-name INSTEAD OF
database-event ON [database-name.] view-name
trigger-action
The database-event can be DELETE, INSERT, UPDATE, or UPDATE OF column-list, any of which can referencethe before and after values of a column using OLD.column-name and NEW.column-name respectively.
The syntax of trigger-action is as follows:
[ FOR EACH ROW | FOR EACH STATEMENT ] [ WHEN expression ]
BEGIN
trigger-step ; [ trigger-step ; ]*
END
A TRigger-step can be any DELETE, INSERT, SELECT, or UPDATE statement or the special RAISE function tocause an exception to be raised in the SQL statement that caused the trigger to fire. The syntax for RAISE is asfollows:
RAISE ( ABORT, error-message ) |
RAISE ( FAIL, error-message ) |
RAISE ( ROLLBACK, error-message ) |
RAISE ( IGNORE )
To drop a trigger use DROP TRIGGER.
DROP TRIGGER [database-name.] trigger-name
The SELECT Statement
To query records from a database use the SELECT statement, which has the following syntax:
SELECT [ALL | DISTINCT] result [FROM table-list]
[WHERE expr]
[GROUP BY expr-list]
[HAVING expr]
[compound-op select]*
[ORDER BY sort-expr-list]
[LIMIT integer [( OFFSET | , ) integer]]
The result takes the form of a list of one or more comma-separated columns, optionally prefixed with the table nameor table alias to avoid ambiguity. A column alias can be specified with the AS keyword following a column identifier.The * character can be used in place of a column list to represent every column from the selected tables.
The table-list is either a comma-separated list of table or view names with optional aliases given following thekeyword AS, or another SELECT statement supplied in parentheses. Tables can also be joined to the table-list usingthe following syntax.
The compound-op can be used to connect SELECT queries using set operations performed on the queries on eitherside of the operator. When three or more queries are compounded, they are evaluated in pairs working from left toright.
The operator can be one of the following keywords: UNION, UNION ALL, INTERSECT, or EXCEPT.
UNION
UNION takes the union of both queries, causing the result of both queries to be returned in one operation. Rows thatappear in the result of both queries are returned only once, so the overall result will contain no duplicate rows.
UNION ALL
UNION ALL is the same as UNION except that duplicate rows may appear in the result.
INTERSECT
INTERSECT takes the intersection of the two datasets, causing only rows that appear in both querieseither side ofthe operatorto be returned once.
EXCEPT
EXCEPT takes the difference between the datasets returned by the queries. All rows from the result to the left querywill be returned except those that also appear in the query to the right of the operator.
The INSERT Statement
To insert records into a table with a list of values given as part of the statement, use the following syntax for theINSERT statement:
INSERT [OR conflict-algorithm]
INTO [database-name .] table-name [(column-list)]
VALUES(value-list)
To insert the dataset returned by a SELECT query into a database table, use this syntax for INSERT.
INSERT [OR conflict-algorithm]
INTO [database-name .] table-name [(column-list)]
select-statement
The REPLACE statement is actually an alias for INSERT OR REPLACE provided for convenience. It can beconsidered to have its own syntax, as follows:
REPLACE INTO [database-name.] table-name [( column-list )] select-statement
Refer to the section "Resolving Conflicts" later in this appendix for more information on this behavior.
The UPDATE Statement
To alter records in a database table, use the UPDATE statement with the following syntax:
UPDATE [ OR conflict-algorithm ] [database-name.] table-name
SET assignment [, assignment]*
[WHERE expr]
Each assignment simply takes the form column-name = expr. The expression expr may reference other column values,and all expressions are evaluated before any assignments are made.
When used without a WHERE clause, the update will be performed on every record in the database.
The DELETE Statement
To delete records from a database, use the DELETE statement.
DELETE FROM [database-name .] table-name [WHERE expr]
When used without a WHERE clause, every row in the table will be deleted.
The COPY Statement
To load a large amount of data into a table, use the COPY command.
COPY [ OR conflict-algorithm ] [database-name.] table-name FROM filename
[ USING DELIMITERS delim ]
The filename can be a file in the current directory or can contain a relative or absolute path. Using stdin for thefilename will read data from standard input.
Resolving Conflicts
The ON CONFLICT and OR clauses are both used to define the conflict resolution algorithm to be used when acolumn constraint has been violated.
An ON-CONFLICT clause can be given in the CREATE TABLE and CREATE INDEX statements to specify thedefault behavior for that table. An OR clause forms part of a COPY, INSERT, or UPDATE statement and overridesany ON-CONFLICT clause on the table.
The ON-CONFLICT or OR keyword is simply followed by one of the following keywords to specify the conflictresolution algorithm to be used: ROLLBACK, ABORT, FAIL, IGNORE, or REPLACE.
ROLLBACK
Upon a violation, end the current transaction with a ROLLBACK and abort the current SQL command with a returncode of SQLITE_CONSTRAINT. If there is no explicit transaction, the action is the same as ABORT.
ABORT
Upon a violation, the current SQL command is aborted with return code SQLITE_CONSTRAINT and any changesalready made as part of that command are reversed. However, a ROLLBACK is not issued, so any changes fromprevious commands within the same transaction are committed. This is the default behavior.
FAIL
The SQL command that caused the violation will abort with return code SQLITE_CONSTRAINT; however, anychanges made up to that point will be committed. For example, in an UPDATE operation, records that match theWHERE criteria that SQLite encounters before the row that causes the violation will be updated, but subsequentmatching rows will not be affected.
IGNORE
When a violation occurs, that row is simply ignored and the SQL statement continues executing normally. No error isreturned and the operation will be carried out successfully on every row that does not cause a violation.
REPLACE
Upon violation of a UNIQUE constraint, the record that already exists that prevents the update or insert operationfrom taking place is removedwithout any delete triggers being firedand the SQL statement continues as normal.Therefore an INSERT or UPDATE statement always takes place and no error is returned. If a NOT NULLconstraint is violated and there is no DEFAULT value on the column, the ABORT algorithm is used instead.
Transactions
To begin a transactionan atomic block of statements that alter the databaseuse BEGIN TRANSACTION.
BEGIN [TRANSACTION [name]]
The keyword trANSACTION is optional, and the name argument, if given, is ignored. SQLite does not supportnested transactions.
To end the transaction, use one of the following commands:
COMMIT [TRANSACTION [name]]
ends the transaction with any changes that have been made saved to the database.
ROLLBACK [TRANSACTION [name]]
ends the transaction, discarding any changes made within that transaction.
Attaching to Other Databases
To attach another database file to the current SQLite session, use ATTACH DATABASE.
ATTACH [DATABASE] database-filename AS database-name
The DATABASE keyword is optional. SQLite will search for database-filename in the current working directory if nopath is given. After a database has been successfully attached, its tables can be referenced in SQL asdatabase-name.table-name.
To detach a database file so that it can no longer be accessed by SQLite, use DETACH DATABASE.
DETACH [DATABASE] database-name
Performance Tuning
Use the VACUUM command to clean up your database.
VACUUM [index-or-table-name]
VACUUM copies the named index or table or, if no name is given, the entire database to a temporary area of diskand reloads the original database from that copy. By doing so, free pages in the database file are removed and thedata in the file is made contiguous.
The EXPLAIN command causes SQLite to report back the virtual machine instructions that would be used toexecute the command.
EXPLAIN sql-statement
From the sqlite program, the .explain command can be used to quickly set a suitable output format for the EXPLAINcommand. Note that the command in sql-statement is not actually executed.
Use the PRAGMA command to modify the operation of the SQLite Library and retrieve information about theconnected database.
PRAGMA name [= value]
PRAGMA function(arg)
These are the currently supported pragmas, which are discussed in depth in Chapter 10, "General DatabaseAdministration."
Commentstext included for reference purposes only, and to be ignored by the parserin SQLite can be written in thesingle-line SQL style, prefixed by two hyphens, or the multiple-line C style, inside /* ... */ characters.
-- single line SQL-style comment
/*
multiple line
C-style comment
*/
ANSI SQL Commands and Features Not Supported The SQL language implemented by SQLite is fairly comprehensive; however, a few commands and features of theANSI-92 specification are not available.
ALTER TABLE
SQLite does not allow the schema of a table to be changed, so the ALTER TABLE command is not supported.
To add, remove, or modify columns in a table, you must drop the table and re-create it with the revised schema. Theusual way to do this is with a temporary table to hold the data from the old table while it is being re-created.
COUNT(DISTINCT column-name)
The DISTINCT keyword cannot appear inside a COUNT function. Instead you must use a nested subquery.
SELECT COUNT(DISTINCT mycol) FROM mytable;
becomes
SELECT COUNT(*) FROM (
SELECT DISTINCT mycol FROM mytable
);
GRANT and REVOKE
Because access to SQLite databases takes place at the filesystem level, the only permissions that can be applied arethose available to the underlying operating system. Therefore the GRANT and REVOKE commands are meaninglessfor SQLite.
INSERT, UPDATE, and DELETE on Views
SQLite does not allow write actions to be performed directly on a view, even if there is only one base table in theview. However, a trigger can be created on a view using the INSTEAD OF syntax, in which you can perform theappropriate INSERT, UPDATE, or DELETE on the underlying table(s).
RIGHT OUTER JOIN
LEFT OUTER JOIN is implemented, but not RIGHT OUTER JOIN. Therefore the table order in your queries mustallow outer joins to be performed in this direction.
CHECK and FOREIGN KEY Constraints
Although the SQL syntax allows CHECK and FOREIGN KEY clauses to be included, they are ignored. They maybe implemented in a future version.
Trigger Limitations
SQLite does not support the FOR EACH STATEMENT type of trigger, or INSTEAD OF TRiggers on tables.INSTEAD OF TRiggers can only be used on views.
Nested Transactions
Only one transaction can be active at a time. The name argument to BEGIN TRANSACTION is ignored.
Variable Subqueries
SQLite evaluates subqueries only once and therefore they cannot refer to variables in the main queryalso known ascorrelated subqueries.
Appendix D. PHP Interface Reference This appendix lists the PHP functions that can be used to communicate with a SQLite database.
Further information and examples of usage submitted by users can often be found in the annotated PHP manual at http://www.php.net/manual/en/ref.sqlite.php.
Predefined Constants Functions that return an array of results can take an optional result_type argument to determine what type of array iscreated. These are the valid constants:
SQLITE_ASSOC causes the array to use the string type column name as the array index.
SQLITE_NUM causes the array to use a numerical index starting from zero for each column in the result.
SQLITE_BOTH causes the array to use both string and numerical keys.
If no constant is specified, SQLITE_BOTH is assumed.
Runtime Configuration In php.ini the sqlite.assoc_case value affects the case of column names used for key values in associative arrays. Itcan take the following values:
0 Mixed case
1 Uppercase
2 Lowercase
The default behavior is to use mixed-case keys reflecting the natural case of the column headings. Using a setting of 1or 2 will cause the case of the keys to be converted to uppercase or lowercase respectively.
Using this option incurs a slight performance penalty, but if case-folding of array keys is required, it is much faster todo so at this level than to implement it in your code.
Function Reference In this section, SQLite functions and their results are described and organized by function category. The function isgiven first, followed by its description.
Opens the database specified by filenameeither in the current directory or referenced via a relative or absolutepathand returns a database connection resource.
Opens a persistent connection to the specified database file. To open an in-memory database, use :memory: as thefilename.
Note
When the sqlite_popen() function is run in a web environment, file permissions must allow the web server user to gainread and write access to both the database file itself and its directory so that journal files can be written.
void sqlite_close ( resource dbhandle)
Closes a database connection.
void sqlite_busy_timeout ( resource dbhandle, int milliseconds)
Specifies the duration in milliseconds for which SQLite should wait for a lock to clear on the database file beforefailing with an SQLITE_BUSY return code. The default value is 60 seconds (60,000 milliseconds).
Causes SQLite to execute the given query and returns a seekable result set resource.
Note
The arguments to sqlite_query() and other such functions can be specified in either order for consistency with bothother SQLite interfaces and other PHP extensions. The preferred ordering is to specify the database resource firsttheorder used by other SQLite extensions.
Works similarly to sqlite_query() but returns a result resource that can only be accessed sequentially. Unless randomaccess is required, this function should be used as it provides much better performance.
string sqlite_escape_string ( string item)
Returns a version of item that is safe for storage by SQLite, by doubling up single quotes and encoding characters thatare not binary-safe.
Error Reporting
int sqlite_last_error ( resource dbhandle)
Returns the error code of the last operation performed on database resource dbhandle.
string sqlite_error_string ( int error_code)
Converts the return value from sqlite_last_error() into a human-readable error message string.
Finding Information About a Query
int sqlite_last_insert_rowid ( resource dbhandle)
Returns the most recently assigned autoincrementing value of an INTEGER PRIMARY KEY field in dbhandle.
int sqlite_changes ( resource dbhandle)
Returns the number of rows affected by the most recent UPDATE or DELETE operation on dbhandle.
int sqlite_num_rows ( resource result)
Returns the total number of rows returned in the result resource result.
int sqlite_num_fields ( resource result)
Returns the total number of fields in the query that produced result resource result.
string sqlite_field_name ( resource result, int field_index)
Returns the name of the field at position field_index in result. The leftmost field in the query has field index zero.
Returns the value of the column from result resource result at the current result handle pointer specified by either fieldindex index or column name name.
bool sqlite_next ( resource result)
Advances the result handle pointer for result to the next row.
bool sqlite_has_more ( resource result)
Returns trUE if result has more rows still to be fetched. FALSE indicates that the result handle pointer is at the end ofthe result.
array sqlite_fetch_array ( resource result
[, int result_type [, bool decode_binary]])
Returns an array containing the next row of data from result and advances the result handle pointer. The types of keyvalues for the array are determined by the optional result_type constant, discussed at the beginning of the chapter.
array sqlite_current ( resource result
[, int result_type [, bool decode_binary]])
Returns an array containing the current row of data from result. The result handle pointer is not changed.
string sqlite_fetch_single ( resource result
[, int result_type [, bool decode_binary]])
Returns an array containing only the first row of data from result. This is the most optimal way to process a result ifonly the first record returned is required.
Random Data Access Functions
A buffered, seekable result resource is required for these functions. They cannot be used if the result was returned bysqlite_unbuffered_query().
bool sqlite_seek ( resource result, int rownum)
Moves the result handle pointer for result to the row specified by rownum. Returns trUE on success or FALSE ifrownum does not exist.
bool sqlite_rewind ( resource result)
Moves the result handle pointer to the very first row of result. Equivalent to using sqlite_seek() to row zero.
Registers a user-defined SQL function called function_name that executes the PHP code in the function with the namegiven in the callback argument. The function is available only within the dbhandle resource.
Registers a user-defined aggregating SQL function called function_name that executes step_func once for each row inthe query, and finalize_func once after all rows have been returned.
string sqlite_udf_encode_binary ( string data)
string sqlite_udf_decode_binary ( string data)
Encodes and decodes, respectively, binary data passed as parameters to a user-defined function. These functionsimpact performance and should only be used where a user-defined function is required to handle binary data.
Appendix E. C Interface Reference This appendix lists the C library functions that can be used to communicate with a SQLite database.
The Core API The core interface to the SQLite library is considered to be just three functions that allow you to open and close adatabase and to execute a query using a user-defined callback function. In this section we'll also look at the errorcodes returned from the core API.
Opening and Closing a Database
You can open and close a database as follows:
sqlite *sqlite_open(
const char *dbname,
int mode,
char **errmsg
);
void sqlite_close(sqlite *db);
The return value of sqlite_open() and the argument to sqlite_close() is an opaque sqlite data structure.
typedef struct sqlite sqlite;
Executing a Query
You can execute a query as follows:
int sqlite_exec(
sqlite *db,
char *sql,
int (*xCallback)(void *, int, char **, char **),
void pArg,
char **errmsg
);
The callback function has the following prototype:
int callback(void *pArg, int argc, char **argv, char **columnNames) {
return 0;
}
Error Codes
This section describes SQLite error codes; each error code is followed by a description of its meaning.
#define SQLITE_OK 0 /* Successful result */
Returned if everything worked and there were no errors.
#define SQLITE_ERROR 1 /* SQL error or missing database */
Indicates an error in the SQL statement being executed.
#define SQLITE_INTERNAL 2 /* An internal logic error in SQLite */
Indicates that an internal consistency check within the SQLite library failed. This code will only ever be returned as aresult of a bug, and such occurrences should be reported to the SQLite mailing list.
Indicates that the database file cannot be opened due to insufficient file permissions.
#define SQLITE_ABORT 4 /* Callback routine requested an abort */
This value is returned if the callback function returns a non-zero value.
#define SQLITE_BUSY 5 /* The database file is locked */
Indicates that another program or thread has the database locked. Only one thread can open a database file forwriting at a time, though multiple reads can take place simultaneously.
#define SQLITE_LOCKED 6 /* A table in the database is locked */
Indicates that the database is locked by a recursive call to sqlite_exec(). Calls to sqlite_exec() from within a callbackfunction are permitted as long as they do not attempt to write to the same table.
#define SQLITE_NOMEM 7 /* A malloc() failed */
This value is returned if a call to malloc() fails due to not enough system memory being available.
#define SQLITE_READONLY 8 /* Attempt to write a readonly database */
Indicates that an attempt was made to write to a database file that was opened in read-only mode.
#define SQLITE_INTERRUPT 9 /* Operation terminated by sqlite_interrupt() */
This value is returned if a database operation is interrupted by the sqlite_interrupt() function being called.
#define SQLITE_IOERR 10 /* Some kind of disk I/O error occurred */
This value is returned if the operating system fails to perform a disk I/O operation, for instance if there is no space lefton the device.
#define SQLITE_CORRUPT 11 /* The database disk image is malformed */
This value would only be returned as a consequence of an unknown error in SQLite or a hardware oroperating-system malfunction. It may indicate that the database file has become corrupted, or that a disk I/O error hasforced SQLite to leave the database file in a corrupted state.
#define SQLITE_NOTFOUND 12 /* (Internal Only) Table or record not found */
For internal use only. This value will never be returned by sqlite_exec().
#define SQLITE_FULL 13 /* Insertion failed because database is full */
This value is returned if an insert operation fails because the database is too big to take any more information, forinstance when the size of the database file would exceed the operating system's file size limit.
#define SQLITE_CANTOPEN 14 /* Unable to open the database file */
This value indicates that that database file could not be opened for some other reason that does not have its ownreturn code.
For internal use only. This value will never be returned by sqlite_exec().
#define SQLITE_SCHEMA 17 /* The database schema changed */
Indicates that another process has altered the database schema since it was first read into memory when the databasewas initially opened. SQLite will re-read the schema when this happens but will not be able to complete the SQLstatement that was being processed. Submitting the statement a second time will usually allow the command to beexecuted.
#define SQLITE_TOOBIG 18 /* Too much data for one row of a table */
Indicates that the maximum row size for a table would be exceeded. The limit is defined at compile time by theMAX_BYTES_PER_ROW value in sqliteInt.h.
#define SQLITE_CONSTRAINT 19 /* Abort due to contraint violation */
Indicates that the SQL statement would have violated a database constraint.
#define SQLITE_MISMATCH 20 /* Data type mismatch */
This value is returned when an attempt is made to insert non-integer data into an INTEGER PRIMARY KEYcolumn.
#define SQLITE_MISUSE 21 /* Library used incorrectly */
Indicates that the SQLite library has been misused in some way, for instance calling sqlite_exec() after the databasehas been closed, or calling sqlite_step() after a SQLITE_DONE or SQLITE_ERROR return code.
#define SQLITE_NOLFS 22 /* Uses OS features not supported on host */
This value is returned if an attempt is made to access a database file larger than 2GB in size on a legacy operatingsystem that does not include large file support.
#define SQLITE_AUTH 23 /* Authorization denied */
Indicates that the authorizer callback has disallowed the SQL you are attempting to execute.
The Non-Callback API The non-callback API provides an alternative way to retrieve data from a SQLite database by compiling an SQLstatement into a virtual machine of type sqlite_vm:
typedef struct sqlite_vm sqlite_vm;
Creating a Virtual Machine
You can create a SQLite virtual machine as follows:
int sqlite_compile(
sqlite *db, /* The open database */
const char *zSql, /* SQL statement to be compiled */
const char **pzTail, /* OUT: uncompiled tail of zSql */
sqlite_vm **ppVm, /* OUT: the virtual machine to execute zSql */
char **pzErrmsg /* OUT: Error message. */
);
The return code from sqlite_compile() is SQLITE_OK if the operation is successful; otherwise, one of the errorcodes listed in the preceding example is returned.
Step-by-Step Execution of an SQL Statement
Each invocation of sqlite_step() for a virtual machine, except the last one, returns a single row of the result:
int sqlite_step(
sqlite_vm *pVm, /* The virtual machine to execute */
int *pN, /* OUT: Number of columns in result */
const char ***pazValue, /* OUT: Column data */
const char ***pazColName /* OUT: Column names and datatypes */
);
int sqlite_finalize(
sqlite_vm *pVm, /* The virtual machine to be finalized */
char **pzErrMsg /* OUT: Error message */
);
Return Codes
The return code from sqlite_step() can be SQLITE_BUSY, SQLITE_ERROR, SQLITE_MISUSE, or either of thefollowing.
#define SQLITE_ROW 100 /* sqlite_step() has another row ready */
Indicates that another row of result data is available.
#define SQLITE_DONE 101 /* sqlite_step() has finished executing */
Indicates that the SQL statement has been completely executed and sqlite_finalize() should now be called.
The return code from sqlite_finalize() indicates the overall success of the SQL command and will be the same as ifthe query had been executed using sqlite_exec().
The Extended API The extended API provides a range of non-core functions to assist with development of software that uses anembedded SQLite database.
Finding Information About the SQLite Library
const char sqlite_version[];
Contains the current library version number.
const char sqlite_encoding[];
Contains the current library encoding version.
Finding Information About Changes to the Database
Several functions can return information about changes that have been made to the database.
int sqlite_last_insert_rowid(sqlite *db);
Returns the most recently assigned autoincrementing value of an INTEGER PRIMARY KEY field.
int sqlite_changes(sqlite *db);
Returns the number of rows affected by an UPDATE or DELETE statement.
Checking SQL Statements
int sqlite_complete(const char *sql);
Returns trUE if a complete SQL statement is provided, that is, the statement ends with a semicolon. Returns FALSEif more characters are required.
Interrupting an SQL Statement
void sqlite_interrupt(sqlite *db);
Causes the current database operation to exist at the first opportunity, returning SQLITE_INTERRUPT to the callingfunction.
Convenience Functions
The following function fetches the entire result of a database query with a single function call:
int sqlite_get_table(
sqlite *db,
char *sql,
char ***result,
int *nrow,
int *ncolumn,
char **errmsg
);
The result will be an array of string pointers containing one element for each column of each row in the result. The firstncolumn elements contain the column names returned. Use nrow and ncolumn to determine which elements of thearray correspond to which values.
The return code from sqlite_get_table() is the same as if sqlite_exec() had executed the query.
void sqlite_free_table(char **azResult);
Frees memory allocated by sqlite_get_table() when it is no longer required.
The _printf() Wrapper Functions
char *sqlite_mprintf(const char *zFormat, ...);
Works like sprintf() but also allows the format strings %q and %Q to manipulate strings for database storage. %qescapes any single quotes by doubling the quote character; %Q additionally encloses the result string within singlequotes.
int sqlite_exec_printf(
sqlite *db,
char *sqlFormat,
int (*)(void *, int, char **, char **),
void *pArg,
char **errmsg,
...
);
Combines sqlite_exec() with sqlite_mprintf(). The format string references items from the sixth argument onwards.
int sqlite_get_table_printf(
sqlite *db,
char *sql,
char ***result,
int *nrow,
int *ncolumn,
char **errmsg,
...
);
Combines sqlite_get_table() with sqlite_mprintf(). The format string references items from the seventh argumentonwards.
Memory Management
void sqlite_freemem(void *p);
Where the library allocates memory using malloc() and returns a pointer to that data, such as errmsg or the result ofsqlite_mprintf(), it is the responsibility of the calling program to free that memory.
Dealing with Locked Database Files
void sqlite_busy_timeout(
sqlite *db,
int ms
);
Specifies an amount of time in milliseconds for which SQLite will wait for a file lock to clear before returningSQLITE_BUSY.
void sqlite_busy_handler(
sqlite *db,
int (*xBusy)(void *, const char *, int),
void *pArg
);
Specifies an alternative busy handler function xBusy. A non-zero response from the handler will cause SQLite toretry; a zero response causes SQLITE_BUSY to be returned.
Performing Background Jobs During Large Queries
void sqlite_progress_handler(
sqlite *db,
int nOps,
int (*xProgress)(void *),
void *pArg
);
Registers a callback routine xProgress to be invoked every nOps virtual machine operations during long-running queryexecution.
Adding New SQL Functions SQLite allows you to add new functions to the SQL language that can subsequently be used in your queries.
Registering Functions
int sqlite_create_function(
sqlite *db,
const char *zName,
int nArg,
void (*xFunc)(sqlite_func*,int,const char**),
void *pUserData
);
Creates a regular function in SQL from the function pointed to by xFunc.
int sqlite_create_aggregate(
sqlite *db,
const char *zName,
int nArg,
void (*xStep)(sqlite_func*,int,const char**),
void (*xFinalize)(sqlite_func*),
void *pUserData
);
Creates an aggregating function with function xStep executed once for each row returned by the query, and xFinalizeinvoked once after all rows have been returned.
The xFunc and xStep arguments are pointers to functions with the following prototype.
void xFunc(
sqlite_func *context,
int argc,
const char **argv
);
The finalize function requires only the context argument.
void xFinalize(sqlite_func *context);
The context argument is an opaque data type sqlite_func.
typedef struct sqlite_func sqlite_func;
Setting Return Values
char *sqlite_set_result_string(
sqlite_func *p,
const char *zResult,
int n
);
void sqlite_set_result_int(
sqlite_func *p,
int iResult
);
void sqlite_set_result_double(
sqlite_func *p,
double rResult
);
Use the appropriate function for the data type that is to be returned to SQL.
void sqlite_set_result_error(
sqlite_func *p,
const char *zMsg,
int n);
Returns an error code to SQLite.
The integer n parameter to sqlite_set_result_string() and sqlite_set_result_error() is the number of characters to bereturned. A negative value will return up to and including the first \0 character.
Referencing Arbitrary Data
void *sqlite_user_data(sqlite_func *p);
Returns the pUserData pointer given in the corresponding sqlite_create_function() or sqlite_create_aggregate() call.
void *sqlite_aggregate_context(
sqlite_func *p,
int nBytes
);
Allocates memory that is unique to a particular instance of the SQL function being called. Memory allocated isautomatically cleaned up when the finalize function is invoked.
Appendix F. Perl Interface Reference This appendix is a reference for the Perl DBI module, which can be used to communicate with a SQLite database.Methods and attributes specific to the DBD::SQLite module are also listed.
In this appendix, the commands or methods are shown first, followed by the explanation.
The Perl DBI The Perl interface to SQLite is the Perl Database Interface (DBI) module using the SQLite Database Driver (DBD)module.
use DBI;
Loads the DBI module into your script. There is no need to explicitly load a DBD driver as DBI takes care of thisautomatically.
In this appendix we use $dbh to refer to a database handle object and $sth for a statement handle.
Opens a database connection using the arguments given and creates a database handle object $dbh.
The data_source argument for a SQLite connection takes the following form, where dbfile can refer to a file in thecurrent working directory or may contain a relative or absolute path.
DBI:SQLite2:dbname=dbfile
The username and auth arguments are not required for a SQLite database and can be omitted or passed as emptystrings.
The optional attr argument can be used to set certain database handle attributes, described later in this appendix.
$rc = $dbh->disconnect();
Closes an open database connection, returning a Boolean $rc value.
Executing SQL Statements
The following methods can be called on a statement handle:
$sth = $dbh->prepare($sql);
$sth = $dbh->prepare($sql, \%attr);
Prepares a statement, $sql, for execution by the database and returns a statement handle object, $sth.
The optional attr argument can be used to set certain statement handle attributes, described later in this appendix.
$sth->execute();
$sth->execute(@bind_values);
Executes a prepared SQL statement. Return value $rv will be the number of rows affected, or -1 if not known. If zerorows are affected, the return value is 0E0, which is treated as a zero value but regarded as true. A zero return codeindicates statement failure.
$dbh->do($sql);
$dbh->do($sql, @bind_values);
A convenience function that prepares and executes a query in one step. The return value is the same as for execute().
Using Bind Values
The optional bind_values list in execute() and do() can be used to specify a comma-separated list of values forplaceholders in the SQL statement. Upon execution, question mark characters in the SQL are replaced with the bindvalues in order. The underlying function called to bind a value is bind_param().
$sth->bind_param($p_num, $bind_value)
$sth->bind_param($p_num, $bind_value, $bind_type)
Parameter number $p_num is replaced with $bind_value and an optional data type hint can be given as $bind_type.Bind parameters are numbered starting at 1.
Making Data Safe
Data inserted using bind parameters is automatically safe; however, to ensure that data from static parameters will beinserted into the database correctly, some delimiting may be necessary.
$dbh->quote($str);
Returns $str enclosed in single quotes, with any awkward characters correctly delimited.
Getting Information About a Query
The rows() method can be called on a statement handle:
$dbh->rows();
Returns the number of rows affected by the most recent UPDATE or DELETE statement.
The last_insert_rowid private method from DBD::SQLite returns the last assigned value of an autoincrementingINTEGER PRIMARY KEY field. It is invoked as follows:
$dbh->func("last_insert_rowid");
Transactions
The following methods can be called on a database handle:
$dbh->begin_work();
Begins a transaction. Equivalent to issuing the SQL command BEGIN TRANSACTION.
$dbh->commit();
Ends a transaction, saving any changes that have been made to the database during the current transaction. Equivalentto issuing the SQL command COMMIT TRANSACTION.
$dbh->rollback();
Ends a transaction without saving the changes made during the current transaction. Equivalent to issuing the SQLcommand ROLLBACK TRANSACTION.
Fetching Rows from a Query
The following methods can be called on a statement handle:
$sth->fetchrow_arrayref();
$sth->fetchrow_array();
$sth->fetchrow_hashref();
$sth->fetchrow_hashref($name);
Fetches the next row of data and returns a data structure holding the field values. The data structure is an arrayreference, list, or hash reference respectively. NULL values are returned as undef.
If there are no more rows to fetch or an error occurs, the return value will be undef for fetchrow_arrayref() andfetchrow_hashref(), and an empty list for fetchrow_array().
The optional $name specifies the statement handle attribute that will be used as the key name in the hash returned byfetchrow_hashref(). The default is NAME, with NAME_uc and NAME_lc used to force the case of the key name.
$sth->fetchall_arrayref();
$sth->fetchall_arrayref($slice);
$sth->fetchall_arrayref($slice, $max_rows);
Returns an array containing the entire dataset from the prepared statement handle.
The optional $slice argument can limit which columns become part of the array using a list of column numbers startingfrom zero. Negative numbers can be used to count from the end of the column list. If no slice is required the argumentcan be undef.
The optional $max_rows argument can be used to limit the number of rows copied into the array.
$sth->fetchall_hashref($key_field);
Returns a hash that contains one entry per row for the entire dataset. The $key_field argument specifies the columnname that is to be used as the key field in the hash.
$sth->finish();
Frees resources associated with a statement resource after it is finished with them.
Error Reporting
The following methods can be called on a database handle to retrieve information about the most recent error.
$dbh->err();
Returns the error code number. For a list of error code values, refer to Appendix E, "C/C++ Interface Reference."
$dbh->errstr();
Returns a string containing a description of the error.
Automatic error checking is activated using the PrintError and RaiseError database attributes, which pass the errormessage to warn() and die() respectively whenever an error is encountered.
$dbh->{PrintError} = 1;
Enables automatic error reporting via warn(). Program execution will continue after an error is reported.
$dbh->{RaiseError} = 1;
Enables automatic error reporting via die(). Program execution will terminate when the error occurs.
$dbh->trace($level);
$dbh->trace($level, $filename);
Sets the trace level for DBI methods performed on that database handle and an optional output filename. Level canbe 1 to 15, with 1 and 2 being most useful. Level zero turns tracing off.
Tracing can also be enabled via the DBI_TRACE environment variable.
DBI_TRACE=1 perl myscript.pl
Runs myscript.pl with trace output at level 1 sent to stdout.
DBI_TRACE=2=trace.log perl myscript.pl
Runs myscript.pl with trace output at level 2 sent to trace.log.
Creating User-Defined Functions
The following calls can be made to private methods implemented by DBD::SQLite:
Registers a new function, $name, in the SQL language that takes $argc arguments and calls the Perl functionreferenced by $func_ref. If $argc is -1, the SQL function will accept any number of arguments.
Registers a new aggregating function $name in the SQL language. $pkg is a package that implements the aggregatorinterface, comprising three methods:
new() is called once and returns the object on which the other methods should be run.
step() is called once for each row that is to be aggregated.
finalize() is called after all the rows have been passed to step() and returns the aggregated result.
Appendix G. Tcl Interface Reference This appendix lists the Tcl commands that are used to communicate with a SQLite database.
The Tcl Library The Tcl library for SQLite is called libtclsqlite.so on Unix platforms and sqlite.dll on Windows systems. The libraryfile should reside in a location that your scripts will be able to load from. Typically this would be a subdirectory of/usr/share/tcl on Unix or C:\tcl\lib on Windows.
In this appendix, commands from the Tcl library are shown first, followed by the explanation.
package require sqlite
Imports the sqlite package into a Tcl script.
Opening and Closing a Database
sqlite dbcmd database-name
Opens a database called database-nameeither from the current directory or referenced via a relative or absolutepathcreating it if it does not already exist. On success, a new command called dbcmd is registered in Tcl, upon whichthe methods described in the rest of this appendix can be applied.
dbcmd close
Closes the database connection and destroys dbcmd.
Executing a Query
dbcmd eval query
Causes SQLite to execute the given query and returns the entire data set fetched as a single list.
dbcmd eval query array { code-block }
As SQLite executes the query, for each row fetched an element in array is created for every column returned by thequery and the commands in code-block are executed.
Additionally the data types of the returned columns are stored as elements named typeof:column-name in array, andthe list of columns returned can be found in array(*).
dbcmd eval query {} { code-block }
If the empty string is used in place of array, code-block is still executed once for each row in the dataset; however,the fetched columns are stored to scalar variables with the same name as their respective columns.
Convenience Functions
dbcmd onecolumn query
Causes SQLite to execute the given query as eval; however, it returns only the first column from the first row of thedataset.
Finding Information About a Query
dbcmd last_insert_rowid
Returns the most recently assigned auto-incrementing value of an INTEGER PRIMARY KEY field following anINSERT operation on dbcmd.
dbcmd changes
Returns the number of rows affected by the most recent UPDATE or DELETE operation on dbcmd or the number ofrows inserted by an INSERT statement.
Checking SQL Statements
dbcmd complete query
Returns true if a complete SQL statement is provided, that is, the statement ends with a semicolon in the appropriateplace. Returns false if more characters are required to complete the statement.
Dealing with Locked Database Files
dbcmd timeout ms
Specifies the duration in milliseconds for which SQLite should wait for a lock to clear on the database file whenperforming a write operation. The default value is zero, meaning it will not wait or retry if the file is locked.
dbcmd busy callback
Specifies a callback function to be executed in the event that a database lock cannot be obtained for a writeoperation. The callback should return zero if you want SQLite to continue trying to get a lock, or non-zero to interruptthe SQL statement.
Error Reporting
dbcmd errorcode
Returns the numeric error code that resulted from the most recent SQLite operation. The full list of error code valuescan be found in Appendix E, "C/C++ Interface Reference."
Finding Information About SQLite
The sqlite_version function will return the version of the tclsqlite library in use.
sqlite> select sqlite_version(*);2.8.13
Custom Functions (UDF)
dbcmd function tcl-func sql-func
Registers a user-defined SQL function called sql-func that executes the Tcl code in the function with the nametcl-func. The function is available only within that instance of dbcmd.
Appendix H. Python Interface Reference This appendix is a reference for the PySQLite extension, which provides an interface to SQLite that is compliant withthe Python Database Specification 2.0.
Before attempting any database connectivity with PySQLite, you must import the sqlite module, using
import sqlite
In this appendix, the commands are shown first, followed by the explanation.
Opening and Closing a Databasecx = sqlite.connect(database, mode=0755, converters={}, autocommit=0,
Opens a SQLite connection to database and returns a connection object to cx.
The mode argument is currently ignored but may be used in the future to specify the file mode with which thedatabase file is opened.
The converters argument can be used to specify mappings from SQL data types using Python conversion functions.
Set autocommit to 1 if you want PySQLite to commit each SQL command immediately when executed, rather thanthe default behavior of batching groups of INSERT, UPDATE, and DELETE statements into a single transaction.
The encoding argument allows you to specify the encoding to be used on Unicode strings. Its value can be noneifonly ASCII characters are to be acceptedor utf-8.
The default timeout value of None means that if the database file is locked, SQLite will return an error immediately.Setting a value in seconds instructs the database engine to keep trying to obtain a lock for the given amount of time.
Use command_logfile with a file object argument to specify a file to which all SQL statements executed throughPySQLite will be written.
cx.close()
Closes a database connection and rolls back any uncommitted transactions. The connection object is unusable after.close() has been called.
Executing SQL Statements
All SQL statements must be executed through a cursor object.
cu = cx.cursor()
Creates a new cursor object for connection cx.
cu.execute(sql)
cu.execute(sql, args)
Passes sql to SQLite for execution. Formatted strings are permitted in sql using the standard Python format codes.Unsafe characters do not require delimiting if passed as format arguments.
cx.commit()
Issues a COMMIT TRANSACTION command to store any unsaved changes to the database. A .commit()operation is always required to save changes when autocommit is off.
cx.rollback()
Issues a ROLLBACK TRANSACTION command to reject any unsaved changes.
numrows = cu.rowcount
Following an operation that changes the database, cu.rowcount returns the number of affected rows. Following aSELECT statement, cu.rowcount returns the number of records in the dataset.
row = cu.fetchone()
Fetches the next data record and advances the cursor to the next row. Returns None if there are no more rows toselect.
rows = cu.fetchmany(num)
Fetches the next num data records and advances the cursor row number by num.
rows = cu.fetchall()
Fetches all remaining data records from a cursor.
rownum = cu.rownumber
Returns the current cursor row position.
Setting Data Type Mappings
The default mappings from SQLite data types to Python are as shown in Chapter 9, "The Python Interface," in Table9.2.
cu.execute("-- types col1, col2, ...")
Specifies the expected data types for col1, col2, and so forth on the next query executed on cursor cu.
Registers a user-defined function called name in SQL using the Python function func. The number of arguments to thefunction is given in argc.
cx.create_aggregate(name, argc, aggregate)
Registers a user-defined aggregating function called name in SQL using the Python class aggregate, which mustcontain the member functions reset(), step(), and finalize().
Error Handling PySQLite extends the StandardError class with its own Error class, with which errors related to SQLite operationscan be handled.
Two subclasses of Error are implementedInterfaceError and DatabaseError. These allow you to detect whether theerror originates with the SQLite engine or the PySQLite interface.
The DatabaseError class is further subdivided as follows:
DataError
OperationalError
IntegrityError
InternalError
ProgrammingError
NotSupportedError
Appendix I. The Future of SQLite This appendix looks at how SQLite might be improved, enhanced, or extended in the future.
SQLite Version 3.0 At the time of writing, SQLite version 3.0 was close to release, and it may very well be available as a stable releaseby the time you are reading this book.
The decision was made to cover the latest 2.8 version throughout the book primarily because of the vast user basethis version of SQLite already has.
Although version 3.0 adds some exciting functionality, it also has a completely different API, so existing users will beslow to migrate, if they switch to the new version at all. New users should consider carefully whether they want to usethe new version or stick with a tried and tested library for their application.
New features will no longer be added to SQLite 2.8, but it will continue to be supported and have maintenance fixesissued for the foreseeable future, so do not be put off from using it simply because a new version has been released.
Let's take a look at some of the changes in SQLite version 3.0.
Naming Changes
Because it is important that SQLite 2.8 can continue to be used while the new version is introduced, version 3.0 usesa new naming convention and allows both versions of the SQLite library to be linked to the same program if required.
The library itself has been renamed to libsqlite3.so on Unix and sqlite3.dll on Windows, and the include file is nowcalled sqlite3.h. The sqlite command-line tool has also been renamed to sqlite3.
Within the API itself, function names have also been changed to be prefixed with sqlite3, but the names themselvesremain familiar; for example, sqlite3_open() and sqlite3_changes() will behave as expected.
Although most of the sqlite3 prefixed function names have the same prototype as their corresponding functions inSQLite 2, one significant difference is the sqlite3_open() function, which works as follows:
int sqlite3_open(const char *filename, sqlite3 **ppDb)
In SQLite 3, the return value from the database connection function is SQLITE_OK on success or the correspondingerror code on failure. The database handle is returned in *ppDb.
The full list of version 3 API calls can be found at http://www.sqlite.org/capi3ref.html.
File Format Changes
The database file format has been overhauled. Version 2.8 databases cannot be read by the 3.0 library, and thereverse is true.
The new file format uses a B+Tree data structure for table storage to replace the B-Trees, which allows betterscalability within a filesystem-based database. It is also more highly optimized through omitting unused fields from thedisk image and better encoding of floating-point numbers, which has been shown to produce a 2535% reduction inthe overall file size.
Migrating from the old database file format to SQLite 3.0 is very simple if you have both the sqlite and sqlite3command-line tools installed. The following command would effectively upgrade olddb to the new file format, savingthe new version as newdb.
$ sqlite olddb .dump | sqlite3 newdb
Data Typing
The typeless nature of SQLite columns has been altered in order that BLOBs (Binary Large Objects) can besupported.
Though any value can still be stored in a column of any declared data type, SQLite version 3.0 assigns each valuestored a particular data type class. The library receives all values as strings; however, if a string appears to containonly a number, it may be considered an INTEGER or NUMERIC, depending on whether or not the number has afractional part.
The data type classes in SQLite 3.0 are
NULL
TEXT
REAL
INTEGER
BLOB
The type of a value is associated with that value itself and not the column in which it is stored. This is sometimescalled manifest typing and is not found in other SQL engines, where the declared column data type determines thetype of all the values stored in it.
SQLite supports the concept of type affinity on columns, where the declared data type is the recommended type forvalues that are inserted into that column. Significantly, the declared types are only recommendations and notrequirements, and it is still possible to insert any kind of data into any column. However, the declared column type willsometimes influence the manifest type of a value.
A column with TEXT affinity will store all data using only the NULL, TEXT, and BLOB classes. A numeric value willbe converted to TEXT before storage. If the column type definition contains CHAR, CLOB, or TEXT, the columnwill be given TEXT affinity.
If the column type definition contains BLOB or no data type is given in the schema, the column will have affinity toNONE. This property ensures that no attempt at conversion will be made on inserted values.
An INTEGER or NUMERIC column can use any of the storage classes, but attempts to convert the value to aninteger or real number before resorting to a TEXT or BLOB type. A decimal number inserted into a column withINTEGER affinity will cause the fractional part to be discarded and the number to be stored as an integer data type.
If the column type definition contains INT, its affinity will be INTEGER; otherwise, it will be NUMERIC.
Tables created with the CREATE TABLE ... AS SELECT syntax will have no data type definitions. All columns willhave affinity NONE.
A few special rules apply when comparing values of different data types. Although INTEGER and NUMERIC typesare compared numerically as you would expect, an INTEGER or NUMERIC type is always considered to be lessthan any TEXT or BLOB regardless of the actual values involved. Any TEXT type is also always less than anyBLOB. When a TEXT or BLOB value is compared to a value of the same type, memcpy() is used to evaluate whichis the greater value. A NULL is always considered less than a value of any other type.
SQLite 3.0 also features a strict affinity mode, whereby if a data item inserted into a column requires the conversionjust described, the database engine will raise an error and perform a rollbackthe behavior that is expected of mostother database engines. There is also no affinity mode, which causes SQLite 3 to handle data types in the samemanner as SQLite 2.
User-Defined Collating Sequences
A collating sequence is a definition of the way in which text strings are ordered. SQLite 3.0 allows you to implementuser-defined collating sequences as functions, which can then be used to determine the sort order within SQLite.
A collating sequence function takes two input arguments and compares them in some custom way. A negative, zero,or positive return value indicates, respectively, that the first value was less than, equal to, or greater than the second.
The COLLATE clause in SQL can appear in the CREATE TABLE statement to define the default collatingsequence for a column or in the ORDER BY clause of a SELECT statement.
Improved Concurrency
The new version of SQLite supports better concurrent use through a more complex locking and journaling schemethat allows reader and writer processes to coexist in the same database. The new mechanism also adds the ability toperform atomic transactions across different attached database files.
Rather than being simply locked when a write operation takes place, the database file can now be in any of thefollowing locking states:
UNLOCKED
SHARED
RESERVED
PENDING
EXCLUSIVE
The UNLOCKED state means that there are no locks on the database and therefore no read or write operationsmay be performed.
Many simultaneous reader processes can all hold SHARED locks on the same database; however, no writeoperation can take place when there is a SHARED lock.
When a process is planning to perform a write, it locks the file as RESERVED. There can be only one RESERVEDlock; however, it can coexist with SHARED locks and new SHARED locks can even be acquired while the databaseis RESERVED. The implication of this is that a writer process can coexist with a readersomething that SQLite 2 wasnot capable of.
The PENDING lock means that a process needs to write to the database as soon as possible. It will already haverequested a RESERVED lock and will be waiting for any SHARED locks to clear before it can perform its write.When a PENDING lock is active, no further SHARED locks can be acquired by other processes. This implements apriority system that ensures new reader processes do not lock the database when a writer is waiting, whereas inSQLite 2 it was possible that a constant stream of reader processes could prevent any writes from taking place.
The only way a write operation can take place is through an EXCLUSIVE lock on the database file. Equivalent tothe writer lock in SQLite 2, no other locks can coexist with an EXCLUSIVE. Through the use of RESERVED andPENDING locks, SQLite keeps the amount of time for which an EXCLUSIVE lock is held to a minimum.
