Basic Module Usage Psycopg 2.5

8/13/2019 Basic Module Usage Psycopg 2.5

1/13

Basic module usage

The basic Psycopg usage is common to all the database adapters implementing the DB API

2.0protocol. Here is an interactive session showing some o the basic commands!

>>> importpsycopg2

# Connect to an existing database

>>> conn = psycopg2.connect("dbname=test user=postgres")

# Open a cursor to perform database operations

>>> cur = conn.cursor()

# Execute a command: this creates a new table

>>> cur.execute("CREATE TABLE test (id serial PRIMARY KEY, num integer, data varc

# Pass data to fill a query placeholders and let Psycopg perform

# the correct conversion (no more SQL injections!)

>>> cur.execute("INSERT INTO test (num, data) VALUES (%s, %s)",

... (100, "abc'def"))

# Query the database and obtain data as Python objects

>>> cur.execute("SELECT * FROM test;")

>>> cur.fetchone()

(1, 100, "abc'def")

# Make the changes to the database persistent

>>> conn.commit()

# Close communication with the database

>>> cur.close()

>>> conn.close()

The main entry points o Psycopg are!

The unction connect()creates a new database session and returns a new connection

instance.

The class connectionencapsulates a database session. It allows to!

create new cursors using the cursor()method to e"ecute database commands

and #ueries$

terminate transactions using the methods commit()or rollback().

The class cursorallows interaction with the database!

send commands to the database using methods such as execute() and

executemany()$

retrieve data rom the database by iteration or using methods such as

fetchone()$ fetchmany()$ fetchall().

Passing parameters to SQL queries

Psycopg casts Python variables to %&' literals by type. (any standard Python types arealready adapted to the correct %&' representation.

)"ample! the Python unction call!

ic module usage Psycopg 2.5.1 documentation http://initd.org/psycopg/docs/usage.html

13 04/07/2013 1:07


2/13

>>> cur.execute(

... """INSERT INTO some_table (an_int, a_date, a_string)

... VALUES (%s, %s, %s);""",

... (10, datetime.date(2005, 11, 18), "O'Reilly"))

is converted into the %&' command!

INSERT INTO some_table (an_int, a_date, a_string)

VALUES (10, '2005-11-18', 'O''Reilly');

*amed arguments are supported too using %(name)splaceholders. +sing named arguments

the values can be passed to the #uery in any order and many placeholders can use the

same values!

>>> cur.execute(

... """INSERT INTO some_table (an_int, a_date, another_date, a_string)

... VALUES (%(int)s, %(date)s, %(date)s, %(str)s);""",

... {'int': 10, 'str': "O'Reilly", 'date': datetime.date(2005, 11, 18)})

,hen parameters are used$ in order to include a literal % in the #uery you can use the %%

string.

,hile the mechanism resembles regular Python strings manipulation$ there are a ew

subtle dierences you should care about when passing parameters to a #uery!

The Python string operator % is not used! the execute() method accepts a tuple or

dictionary o values as second parameter. Neveruse % or + to merge values into

#ueries.

The variables placeholder must always be a%s$ even i a dierent placeholder -such

as a %dor integers or %for loats may loo/ more appropriate!

>>> cur.execute("INSERT INTO numbers VALUES (%d)", (42,)) # WRONG

>>> cur.execute("INSERT INTO numbers VALUES (%s)", (42,)) # correct

or positional variables binding$ the second argument must always be a sequence$

even i it contains a single variable. And remember that Python re#uires a comma to

create a single element tuple!

>>> cur.execute("INSERT INTO foo VALUES (%s)", "bar") # WRONG

>>> cur.execute("INSERT INTO foo VALUES (%s)", ("bar")) # WRONG

>>> cur.execute("INSERT INTO foo VALUES (%s)", ("bar",)) # correct

>>> cur.execute("INSERT INTO foo VALUES (%s)", ["bar"]) # correct

1nly variable values should be bound via this method! it shouldnt be used to set

table or ield names. or these elements$ ordinary string ormatting should be used

beore running execute().

The problem with the query parameters

The %&' representation or many data types is oten not the same o the Python string

representation. The classic e"ample is with single #uotes in strings! %&' uses them as


13 04/07/2013 1:07


3/13

string constants bounds and re#uires them to be escaped$ whereas in Python single #uotes

can be let unescaped in strings bounded by double #uotes. or this reason a na3ve

approach to the composition o #uery strings$ e.g. using string concatenation$ is a recipe or

terrible problems!

>>> SQL = "INSERT INTO authors (name) VALUES ('%s');"# NEVER DO THIS

>>> data = ("O'Reilly", )

>>> cur.execute(SQL % data) # THIS WILL FAIL MISERABLYProgrammingError: syntax error at or near "Reilly"

LINE 1: INSERT INTO authors (name) VALUES ('O'Reilly')

^

I the variable containing the data to be sent to the database comes rom an untrusted

source -e.g. a orm published on a web site an attac/er could easily crat a malormed

string$ either gaining access to unauthori4ed data or perorming destructive operations on

the database. This orm o attac/ is called %&' in5ectionand is /nown to be one o the most

widespread orms o attac/ to servers. Beore continuing$ please print this pageas a memo

and hang it onto your des/.

Psycopg can automatically convert Python ob5ects to and rom %&' literals! using this

eature your code will be more robust and reliable. ,e must stress this point!

Warning: *ever$ never$ NEVERuse Python string concatenation -+ or string

parameters interpolation -% to pass variables to a %&' #uery string. *ot even at

gunpoint.

The correct way to pass variables in a %&' command is using the second argument o the

execute()method!

>>> SQL = "INSERT INTO authors (name) VALUES (%s);"# Note: no quotes

>>> data = ("O'Reilly", )

>>> cur.execute(SQL, data) # Note: no % operator

Adaptation of Python values to SQL types

(any standard Python types are adapted into %&' and returned as Python ob5ects when a

#uery is e"ecuted.

The ollowing table shows the deault mapping between Python and Postgre%&' types!

Python PostgreSQL See alsoNone NULL Constants adaptationbool bool

float real

doubleNumbers adaptation

int

long

smallint

integer

bigint

Decimal numeric

str

unicode

varchar

textStrings adaptation


13 04/07/2013 1:07


4/13

Python PostgreSQL See alsobuffer

memoryview

bytearray

bytes

Buer protocol

bytea Binary adaptation

date date Date/Time objects adaptationtime time

datetime timestamptimestamptz

timedelta interval

list ARRAY Lists adaptationtuple

namedtuple6omposite typesINsynta"

Tuples adaptation

Composite types castingdict hstore Hstore data type

Psycopgs Range range ange data types

Anything7 json !S"N adaptationuuid uuid ##$D data type

The mapping is airly customi4able! see%dapting new &ython types to S'L synta(and Type

casting o) S'L types into &ython objects. 8ou can also ind a ew other speciali4ed adapters

in thepsycopg2.extrasmodule.

Constants adaptation

PythonNoneand boolean values Trueand Falseare converted into the proper %&' literals!

>>> cur.mogrify("SELECT %s, %s, %s;", (None, True, False))'SELECT NULL, true, false;'

umbers adaptation

Python numeric ob5ects int$ long$ float$ Decimal are converted into a Postgre%&'

numerical representation!

>>> cur.mogrify("SELECT %s, %s, %s, %s;", (10, 10L, 10.0, Decimal("10.00")))

'SELECT 10, 10, 10.0, 10.00;'

9eading rom the database$ integer types are converted into int$ loating point types are

converted into float$ numeric:decimalare converted into Decimal.

Note: %ometimes you may preer to receive numericdata as floatinstead$ or

perormance reason or ease o manipulation! you can conigure an adapter to cast

&ostgreS'L numeric to &ython )loat. This o course may imply a loss o precision.

See also: Postgre%&' numeric types

Strings adaptation


13 04/07/2013 1:07


5/13

Python strand unicodeare converted into the %&' string synta". unicodeob5ects -strin

Python ; are encoded in the connection encodingbeore sending to the bac/end! trying to

send a character not supported by the encoding will result in an error. Data is usually

received as str -i*e* it is decodedon Python ;$ let encodedon Python 2. However it is

possible to receive unicodeon Python 2 too! see #nicode handling.

!nicode handling

Psycopg can e"change +nicode data with a Postgre%&' database. Python unicodeob5ects

are automatically encodedin the client encoding deined on the database connection -the

Postgre%&' encoding$ available in connection.encoding$ is translated into a Python codec

using the encodingsmapping!

>>>printu, type(u)

>>> cur.execute("INSERT INTO test (num, data) VALUES (%s,%s);", (74, u))

,hen reading data rom the database$ in Python 2 the strings returned are usually < bit str

ob5ects encoded in the database client encoding!

>>>printconn.encoding

UTF8

>>> cur.execute("SELECT data FROM test WHERE num = 74")

>>> x = cur.fetchone()[0]

>>>printx, type(x), repr(x)

'\xc3\xa0\xc3\xa8\xc3\xac\xc3\xb2\xc3\xb9\xe2\x82\xac'

>>> conn.set_client_encoding('LATIN9')



>>>printtype(x), repr(x)

'\xe0\xe8\xec\xf2\xf9\xa4'

In Python ; instead the strings are automatically decodedin the connection encoding$ as the

strob5ect can represent +nicode characters. In Python 2 you must register a typecasterin

order to receive unicodeob5ects!

>>> psycopg2.extensions.register_type(psycopg2.extensions.UNICODE, cur)



>>>printx, type(x), repr(x)

u'\xe0\xe8\xec\xf2\xf9\u20ac'

In the above e"ample$ the UNICODE typecaster is registered only on the cursor. It is also

possible to register typecasters on the connection or globally! see the unction

register_type()and Type casting o) S'L types into &ython objectsor details.

Note: In Python 2$ i you want to uniormly receive all your database input in +nicode$


13 04/07/2013 1:07


6/13

you can register the related typecasters globally as soon as Psycopg is imported!

importpsycopg2

importpsycopg2.extensions

psycopg2.extensions.register_type(psycopg2.extensions.UNICODE)

psycopg2.extensions.register_type(psycopg2.extensions.UNICODEARRAY)

and orget about this story.

Binary adaptation

Python types representing binary ob5ects are converted into Postgre%&' binary string

synta"$ suitable or bytea ields. %uch types are buffer -only available in Python 2$

memoryview -available rom Python 2.=$bytearray -available rom Python 2.> andbytes

-only rom Python ;! the name is available rom Python 2.> but its only an alias or the type

str. Any ob5ect implementing the 9evised Buer Protocolshould be usable as binary type

where the protocol is supported -i.e. rom Python 2.>. 9eceived data is returned asbuffer

-in Python 2 ormemoryview-in Python ;.

6hanged in version 2.?! only strings were supported beore.

6hanged in version 2.?.@! can parse the he" ormat rom .0 servers without relying on the

version o the client library.

Note: In Python 2$ i you have binary data in a strob5ect$ you can pass them to a bytea

ield using thepsycopg2.Binarywrapper!

mypic = open('picture.png', 'rb').read()

curs.execute("insert into blobs (file) values (%s)",

(psycopg2.Binary(mypic),))

Warning: %ince version .0 Postgre%&' uses by deault a new Che" ormatto emit

byteaields. %tarting rom Psycopg 2.?.@ the ormat is correctly supported. I you use a

previous version you will need some e"tra care when receiving bytea rom Postgre%&'!

you must have at least libp# .0 installed on the client or alternatively you can set the

byteaEoutputconiguration parameter to escape$ either in the server coniguration ile or inthe client session -using a #uery such as SET bytea_output TO escape; beore receiving

binary data.

"ate#Time ob$ects adaptation

Python builtin datetime$ date$ time$ timedelta are converted into Postgre%&'s

timestamp[tz] $ date$ time$ interval data types. Time 4ones are supported too. The

)geni" m".DateTimeob5ects are adapted the same way!

>>> dt = datetime.datetime.now()

>>> dt

datetime.datetime(2010, 2, 8, 1, 40, 27, 425337)


13 04/07/2013 1:07


7/13

>>> cur.mogrify("SELECT %s, %s, %s;", (dt, dt.date(), dt.time()))

"SELECT '2010-02-08T01:40:27.425337', '2010-02-08', '01:40:27.425337';"

>>> cur.mogrify("SELECT %s;", (dt - datetime.datetime(2010,1,1),))

"SELECT '38 days 6027.425337 seconds';"

See also: Postgre%&' date:time types

Time %ones handling

The Postgre%&' type timestamp with time zone -a./.a. timestamptz is converted into

Python datetimeob5ects with a tzinfoattribute set to a FixedOffsetTimezoneinstance.

>>> cur.execute("SET TIME ZONE 'Europe/Rome';") # UTC + 1 hour

>>> cur.execute("SELECT '2010-01-01 10:30:45'::timestamptz;")

>>> cur.fetchone()[0].tzinfo

psycopg2.tz.FixedOffsetTimezone(offset=60, name=None)

*ote that only time 4ones with an integer number o minutes are supported! this is a

limitation o the Python datetimemodule. A ew historical time 4ones had seconds in the

+T6 oset! these time 4ones will have the oset rounded to the nearest minute$ with an

error o up to ;0 seconds.

>>> cur.execute("SET TIME ZONE 'Asia/Calcutta';") # offset was +5:53:20

>>> cur.execute("SELECT '1930-01-01 10:30:45'::timestamptz;")

>>> cur.fetchone()[0].tzinfo

psycopg2.tz.FixedOffsetTimezone(offset=353, name=None)

6hanged in version 2.2.2! time4ones with seconds are supported -with rounding.

Previously such time4ones raised an error. In order to deal with them in previous versions

usepsycopg2.extras.register_tstz_w_secs().

&nfinite dates handling

Postgre%&' can store the representation o an Cininite date$ timestamp$ or interval. Ininite

dates are not available to Python$ so these ob5ects are mapped to date.max$ datetime.max$

interval.max. +nortunately the mapping cannot be bidirectional so these dates will bestored bac/ into the database with their values$ such as 9999-12-31.

It is possible to create an alternative adapter or dates and other ob5ects to map date.maxto

infinity $ or instance!

classInfDateAdapter:

def__init__(self, wrapped):

self.wrapped = wrapped

defgetquoted(self):

ifself.wrapped == datetime.date.max:

return"'infinity'::date"

elifself.wrapped == datetime.date.min:

return"'-infinity'::date"

else:

returnpsycopg2.extensions.DateFromPy(self.wrapped).getquoted()


13 04/07/2013 1:07


8/13

psycopg2.extensions.register_adapter(datetime.date, InfDateAdapter)

1 course it will not be possible to write the value o date.max in the database anymore!

infinity will be stored instead.

Lists adaptation

Python lists are converted into Postgre%&' ARRAYs!

>>> cur.mogrify("SELECT %s;", ([10, 20, 30], ))

'SELECT ARRAY[10,20,30];'

Note: 8ou can use a Python list as the argument o the INoperator using the

Postgre%&' A*8 operator.

ids = [10, 20, 30]

cur.execute("SELECT * FROM data WHERE id = ANY(%s);", (ids,))

urthermore ANYcan also wor/ with empty lists$ whereas IN ()is a %&' synta" error.

Note: 9eading bac/ rom Postgre%&'$ arrays are converted to lists o Python ob5ects as

e"pected$ but only i the items are o a /nown type. Arrays o un/nown types are returned

as represented by the database -e.g. {a,b,c}. I you want to convert the items into

Python ob5ects you can easily create a typecaster or array o) un+nown types.

Tuples adaptation

Python tuples are converted into a synta" suitable or the %&' IN operator and to represent

a composite type!

>>> cur.mogrify("SELECT %sIN %s;", (10, (10, 20, 30)))

'SELECT 10 IN (10, 20, 30);'

Note: %&' doesnt allow an empty list in the INoperator$ so your code should guard

against empty tuples. Alternatively you can use a &ython list.

I you want Postgre%&' composite types to be converted into a Python tuple:namedtuple

you can use the register_composite()unction.

*ew in version 2.0.>! the tuple INadaptation.

6hanged in version 2.0.@?! the tuple INadapter is always active. In previous releases it was

necessary to import the extensionsmodule to have it registered.

6hanged in version 2.;! namedtupleinstances are adapted li/e regular tuples and can thus

be used to represent composite types.


13 04/07/2013 1:07


9/13

Transactions control

In Psycopg transactions are handled by the connectionclass. By deault$ the irst time a

command is sent to the database -using one o the cursors created by the connection$ a

new transaction is created. The ollowing database commands will be e"ecuted in the

conte"t o the same transaction F not only the commands issued by the irst cursor$ but the

ones issued by all the cursors created by the same connection. %hould any command ail$the transaction will be aborted and no urther command will be e"ecuted until a call to the

rollback()method.

The connection is responsible or terminating its transaction$ calling either the commit()or

rollback()method. 6ommitted changes are immediately made persistent into the database.

6losing the connection using the close() method or destroying the connection ob5ect

-using delor letting it all out o scope will result in an implicit rollbac/.

It is possible to set the connection in autocommit mode! this way all the commands

e"ecuted will be immediately committed and no rollbac/ is possible. A ew commands -e.g.

CREATE DATABASE$ VACUUM ... re#uire to be run outside any transaction! in order to be able to

run these commands rom Psycopg$ the connection must be in autocommit mode! you can

use the autocommitproperty -set_isolation_level()in older versions.

Warning: By deault even a simple SELECTwill start a transaction! in longGrunning

programs$ i no urther action is ta/en$ the session will remain Cidle in transaction$ a

condition non desiderable or several reasons -loc/s are held by the session$ tables

bloat.... or long lived scripts$ either ma/e sure to terminate a transaction as soon as

possible or use an autocommit connection.

A ew other transaction properties can be set sessionGwide by the connection! or instance it

is possible to have readGonly transactions or change the isolation level. %ee the

set_session()method or all the details.

withstatement

%tarting rom version 2.$ psycopg2s connections and cursors are conte(t managersand

can be used with the withstatement!

withpsycopg2.connect(DSN) asconn:

withconn.cursor() ascurs:

curs.execute(SQL)

,hen a connection e"its the withbloc/$ i no e"ception has been raised by the bloc/$ the

transaction is committed. In case o e"ception the transaction is rolled bac/. In no case the

connection is closed! a connection can be used in more than a with statement and each

with

bloc/ is eectively wrapped in a transaction.

,hen a cursor e"its the with bloc/ it is closed$ releasing any resource eventually

associated with it. The state o the transaction is not aected.


13 04/07/2013 1:07


10/13

Server side cursors

,hen a database #uery is e"ecuted$ the Psycopg cursor usually etches all the records

returned by the bac/end$ transerring them to the client process. I the #uery returned an

huge amount o data$ a proportionally large amount o memory will be allocated by the

client.

I the dataset is too large to be practically handled on the client side$ it is possible to create

a ser,er sidecursor. +sing this /ind o cursor it is possible to transer to the client only a

controlled amount o data$ so that a large dataset can be e"amined without /eeping it

entirely in memory.

%erver side cursor are created in Postgre%&' using the DECLARE command and

subse#uently handled using MOVE$ FETCHand CLOSEcommands.

Psycopg wraps the database server side cursor in named cursors. A named cursor is

created using the cursor()method speciying the nameparameter. %uch cursor will behavemostly li/e a regular cursor$ allowing the user to move in the dataset using the scroll()

method and to read the data using fetchone()and fetchmany()methods. *ormally you can

only scroll orward in a cursor! i you need to scroll bac/wards you should declare your

cursor scrollable.

*amed cursors are also iterableli/e regular cursors. *ote however that beore Psycopg 2.?

iteration was perormed etching one record at time rom the bac/end$ resulting in a large

overhead. The attribute itersizenow controls how many records are etched at time during

the iteration! the deault value o 2000 allows to etch about @00B per roundtrip assumingrecords o @0G20 columns o mi"ed number and stringsJ you may decrease this value i you

are dealing with huge records.

*amed cursors are usually created WITHOUT HOLD$ meaning they live only as long as the

current transaction. Trying to etch rom a named cursor ater a commit() or to create a

named cursor when the connectiontransaction isolation level is set toAUTOCOMMITwill result

in an e"ception. It is possible to create a WITH HOLDcursor by speciying a Truevalue or the

withholdparameter to cursor()or by setting the withholdattribute to True beore calling

execute()on the cursor. It is e"tremely important to always close()such cursors$ otherwise

they will continue to hold serverGside resources until the connection will be eventually

closed. Also note that while WITH HOLD cursors lietime e"tends well ater commit()$ calling

rollback()will automatically close the cursor.

Note: It is also possible to use a named cursor to consume a cursor created in some

other way than using the DECLAREe"ecuted by execute(). or e"ample$ you may have a

P':pg%&' unction returning a cursor!

CREATE FUNCTION reffunc(refcursor) RETURNS refcursor AS $$

BEGIN OPEN $1 FOR SELECT col FROM test;

RETURN $1;

END;

$$ LANGUAGE plpgsql;


de 13 04/07/2013 1:07


11/13

8ou can read the cursor content by calling the unction with a regular$ nonGnamed$

Psycopg cursor!

cur1 = conn.cursor()

cur1.callproc('reffunc', ['curname'])

and then use a named cursor in the same transaction to Csteal the cursor!

cur2 = conn.cursor('curname')

forrecord incur2: # or cur2.fetchone, fetchmany...

# do something with record

pass

Thread and process safety

The Psycopg module and the connectionob5ects are thread-sa)e! many threads can access

the same database either using separate sessions and creating a connectionper thread or

using the same connection and creating separate cursors. In DB API 2.0parlance$ Psycopg

is le,el . thread sa)e.

The dierence between the above two approaches is that$ using dierent connections$ the

commands will be e"ecuted in dierent sessions and will be served by dierent server

processes. 1n the other hand$ using many cursors on the same connection$ all the

commands will be e"ecuted in the same session -and in the same transaction i the

connection is not in autocommitmode$ but they will be seriali4ed.

The above observations are only valid or regular threads! they dont apply to or/ed

processes nor to green threads. libpq connections shouldnt be used by a or/ed

processes$ so when using a module such asmultiprocessing or a or/ing web deploy

method such as ast6KI ma/e sure to create the connections a)terthe or/.

6onnections shouldnt be shared either by dierent green threads! see Support )or

coroutine librariesor urther details.

!sing C'P( T' and C'P( )*'+Psycopg cursorob5ects provide an interace to the eicient Postgre%&' COPYcommand to

move data rom iles to tables and bac/. The methods e"posed are!

copy_from()

9eads data )rom a ileGli/e ob5ect appending them to a database table -COPY table

FROM filesynta". The source ile must have both read()and readline()method.

copy_to()

,rites the content o a table toa ileGli/e ob5ect -COPY table TO file synta". Thetarget ile must have a write()method.

copy_expert()


e 13 04/07/2013 1:07


12/13

Allows to handle more speciic cases and to use all the COPY eatures available in

Postgre%&'.

Please reer to the documentation o the single methods or details and e"amples.

Access to PostgreSQL large ob$ects

Postgre%&' oers support or large ob5ects$ which provide streamGstyle access to user data

that is stored in a special largeGob5ect structure. They are useul with data values too large

to be manipulated conveniently as a whole.

Psycopg allows access to the large ob5ect using the lobjectclass. 1b5ects are generated

using the connection.lobject()actory method. Data can be retrieved either as bytes or as

+nicode strings.

Psycopg large ob5ect support eicient import:e"port with ile system iles using the

lo_import()and lo_export()libp# unctions.

Two,Phase Commit protocol support

*ew in version 2.;.

Psycopg e"poses the twoGphase commit eatures available since Postgre%&' ? bytes

a branch #ualiier -string not longer than >? bytes

or a particular global transaction$ the irst two components will be the same or all the

resources. )very resource will be assigned a dierent branch #ualiier.

According to the DB API 2.0 speciication$ a transaction ID is created using the

connection.xid()method. 1nce you have a transaction id$ a distributed transaction can be

started with connection.tpc_begin()$ prepared using tpc_prepare() and completed using

tpc_commit()or tpc_rollback(). Transaction IDs can also be retrieved rom the database

using tpc_recover()and completed using the above tpc_commit()and tpc_rollback().

Postgre%&' doesnt ollow the LA standard though$ and the ID or a Postgre%&' prepared

transaction can be any string up to 200 characters long. Psycopgs Xid ob5ects can

represent both LAGstyle transactions IDs -such as the ones created by the xid()method

and Postgre%&' transaction IDs identiied by an unparsed string.

The ormat in which the Lids are converted into strings passed to the database is the same

employed by the Postgre%&' MDB6 driver! this should allow interoperation between tools

written in Python and in Mava. or e"ample a recovery tool written in Python would be able


de 13 04/07/2013 1:07


13/13

to recogni4e the components o transactions produced by a Mava program.

or urther details see the documentation or the above methods.


Date post:	04-Jun-2018
Category:	Documents
Upload:	constantin-constantius
View:	228 times
Download:	0 times

Basic Module Usage Psycopg 2.5

Documents