OverviewMethodology
CodeTest Cases
Results
Postgres Interface Testing
credativ USA
June 10, 2014
Joe Conway AustinPUG
OverviewMethodology
CodeTest Cases
Results
Intro
Motivation
Answer this question:
What is relative performance ofODBC, JDBC, and libpq?
Specifically:
Connection speedQuery result retrievalAlternate retrieval methodsMaterializing to file
Joe Conway AustinPUG
OverviewMethodology
CodeTest Cases
Results
High LevelApproach Detail
Approach
Seed database
Launch command line clients
Vary conditions
Collect durations
Analyze results
Joe Conway AustinPUG
OverviewMethodology
CodeTest Cases
Results
High LevelApproach Detail
Major Components
Command line clients
PG Extension
Driver, load, and result objects
Driver function
Launcher bash script
Joe Conway AustinPUG
OverviewMethodology
CodeTest Cases
Results
High LevelApproach Detail
Flow
Joe Conway AustinPUG
OverviewMethodology
CodeTest Cases
Results
High LevelApproach Detail
Clients
Individual command line clientsNative language for interface
libpq - CODBC - CJDBC - Java
As similar as possible source code structureIdentical command line argumentsHandle variety of test conditions
Joe Conway AustinPUG
OverviewMethodology
CodeTest Cases
Results
High LevelApproach Detail
Extension
C extension for command execution
Simple way to execute commandsComplete control over string escaping, etc.Allow entire test to be table drivenEasy test result (duration) data collection
Joe Conway AustinPUG
OverviewMethodology
CodeTest Cases
Results
High LevelApproach Detail
Driver Tables
alternateclientcopy ⇒ acc-n, acc-a, acc-y
conninfos ⇒ lcl:ipc, lcl:tcp, rmt:tcp
interfaces ⇒ libpq, ODBC, JDBC
printfracdest ⇒ p-all:null, p-none:null, p-all:file
sqlstratiter⇒ sql001:0:50k, sql50k:0:001, sql10m:0:001⇒ sql001:1:50k, sql50k:1:001, sql10m:1:001
Joe Conway AustinPUG
OverviewMethodology
CodeTest Cases
Results
High LevelApproach Detail
Driver View
testcases
Generates all testcases from driver tablesUses both JOIN (inner) and CROSS JOIN (Cartesian product)
iftest=# select * from testcases limit 1;
-[ RECORD 1 ]-----------------------------------------------------
testcase | ODBC:lcl:ipc:sql001:0:50k:acc-n:p-all:null
testcmd | odbctest -c DSN=pgsql_ipc \
-s "select * from t50k where id = 4242" \
-f 1 -k 0 -n 50000 > /dev/null 2>&1
Joe Conway AustinPUG
OverviewMethodology
CodeTest Cases
Results
High LevelApproach Detail
Load Tables
t10m - 10 million rowsiftest=# select * from t10m limit 1;
-[ RECORD 1 ]------------------------------------------
x1 | 1
x2 | 1aaaaaaaaaaaaa
x3 | 2bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
x4 | 3ccccccccccccc
x5 | 4aaaaaaaaaaaaa
x6 | 5bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
x7 | 6ccccccccccccc
x8 | 7aaaaaaaaaaaaa
x9 | 8bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
x10 | 9ccccccccccccc
t50k - 50 thousand rows
iftest=# select * from t50k limit 1;
(similar row but wider text)
Joe Conway AustinPUG
OverviewMethodology
CodeTest Cases
Results
High LevelApproach Detail
Results Table
testresults - aggregate test case representationiftest=# \x
Expanded display is on.
iftest=# select * from testresults limit 1;
-[ RECORD 1 ]----------------------------------------
testcase | JDBC:lcl:tcp:sql001:0:50k:acc-n:p-all:null
iter | 1
ts | 2014-01-20 06:34:09.442483-08
duration | 1.223566
Joe Conway AustinPUG
OverviewMethodology
CodeTest Cases
Results
High LevelApproach Detail
Results View
vtestresults - expanded test case representationiftest=# select * from vtestresults limit 1;
-[ RECORD 1 ]----------------------------
iface | JDBC
loc | lcl
comm | tcp
sql | sql001
connstrat | 0
numiter | 50k
extra | acc-n
prnt | p-all
dest | file
iter | 1
ts | 2014-01-20 06:34:09.442483-08
duration | 0.185309
Joe Conway AustinPUG
OverviewMethodology
CodeTest Cases
Results
High LevelApproach Detail
Driver Function
exectestcases()
Arguments
path - client binary location (command line clients)ldlibpath - linked library location (libpq.so)classpath - Java class location (JDBC and getopt jar files)outeriter - how many times to execute all test cases
Returns SETOF RECORD
iter - current outeritertcase - aggregate form test case identifierdur - duration of test case run
Joe Conway AustinPUG
OverviewMethodology
CodeTest Cases
Results
High LevelApproach Detail
Launcher script
Kicks off the entire automated test
Sets variables for exectestcases() argumentsCalls exectestcases() using psqlCurrently executes 288 test cases per runRuns for many hours and produces many GB output files
Joe Conway AustinPUG
OverviewMethodology
CodeTest Cases
Results
Command LinePG ExtensionDriver, Load, and Result ObjectsDriver FunctionLauncher Script
libpq Client
See libpqtest.c
Joe Conway AustinPUG
OverviewMethodology
CodeTest Cases
Results
Command LinePG ExtensionDriver, Load, and Result ObjectsDriver FunctionLauncher Script
ODBC Client
See odbctest.c, odbc.ini, odbcinst.ini
Joe Conway AustinPUG
OverviewMethodology
CodeTest Cases
Results
Command LinePG ExtensionDriver, Load, and Result ObjectsDriver FunctionLauncher Script
JDBC Client
See jdbctest.java
Joe Conway AustinPUG
OverviewMethodology
CodeTest Cases
Results
Command LinePG ExtensionDriver, Load, and Result ObjectsDriver FunctionLauncher Script
Extension Code
See pgiftest.*, Makefile.ext, Makefile
-- make the extension available
CREATE EXTENSION pgiftest;
Joe Conway AustinPUG
OverviewMethodology
CodeTest Cases
Results
Command LinePG ExtensionDriver, Load, and Result ObjectsDriver FunctionLauncher Script
Object Creation SQL
See test-plan.sql
Joe Conway AustinPUG
OverviewMethodology
CodeTest Cases
Results
Command LinePG ExtensionDriver, Load, and Result ObjectsDriver FunctionLauncher Script
Function SQL
See test-plan.sql
Joe Conway AustinPUG
OverviewMethodology
CodeTest Cases
Results
Command LinePG ExtensionDriver, Load, and Result ObjectsDriver FunctionLauncher Script
bash Script
See runtests.sh
Joe Conway AustinPUG
OverviewMethodology
CodeTest Cases
Results
InterfacesConn TypeSizingPrintExtra
Interface Variations
SELECT * FROM interfaces;
ifname | testprog
--------+---------------
ODBC | odbctest
libpq | libpqtest
JDBC | java jdbctest
(3 rows)
Joe Conway AustinPUG
OverviewMethodology
CodeTest Cases
Results
InterfacesConn TypeSizingPrintExtra
Connection Type Variations
SELECT * FROM conninfos;
conndesc | ifname | conninfo
----------+--------+-------------------------------------------------------
lcl:ipc | ODBC | -c DSN=pgsql_ipc
lcl:tcp | ODBC | -c DSN=pgsql_lcl
rmt:tcp | ODBC | -c DSN=pgsql_rmt
lcl:ipc | libpq | -c postgresql://%2Ftmp/iftest?user=postgres
lcl:tcp | libpq | -c postgresql://localhost/iftest?user=postgres
rmt:tcp | libpq | -c postgresql://192.168.1.3/iftest?user=postgres
lcl:tcp | JDBC | -c jdbc:postgresql://localhost/iftest?user=postgres
rmt:tcp | JDBC | -c jdbc:postgresql://192.168.1.3/iftest?user=postgres
(8 rows)
Joe Conway AustinPUG
OverviewMethodology
CodeTest Cases
Results
InterfacesConn TypeSizingPrintExtra
Result Size, Conn Count, Num Executions
SELECT sqlstratiterdesc as desc,
sql,
connstrat as strat,
numiter
FROM sqlstratiter;
desc | sql | strat | numiter
--------------+-----------------------------------------+-------+----------
sql001:0:50k | -s "select * from t50k where id = 4242" | -k 0 | -n 50000
sql50k:0:001 | -s "select * from t50k" | -k 0 | -n 1
sql10m:0:001 | -s "select * from t10m" | -k 0 | -n 1
sql001:1:50k | -s "select * from t50k where id = 4242" | -k 1 | -n 50000
sql50k:1:001 | -s "select * from t50k" | -k 1 | -n 1
sql10m:1:001 | -s "select * from t10m" | -k 1 | -n 1
(6 rows)
Joe Conway AustinPUG
OverviewMethodology
CodeTest Cases
Results
InterfacesConn TypeSizingPrintExtra
Materialize (Y/N) and Destination
SELECT * FROM printfracdest;
pfddesc | printfrac | dest
-------------+--------------+-----------------
p-all:null | -p 1 | /dev/null
p-all:file | -p 1 | /tmp/iftest.out
p-none:null | -p 0 | /dev/null
(3 rows)
Joe Conway AustinPUG
OverviewMethodology
CodeTest Cases
Results
InterfacesConn TypeSizingPrintExtra
Use Alternate Fetch or COPY Interface
SELECT * FROM alternateclientcopy;
accdesc | ifname | sw
---------+--------+----
acc-n | ODBC |
acc-n | libpq |
acc-a | libpq | -a
acc-y | libpq | -y
acc-n | JDBC |
acc-y | JDBC | -y
(6 rows)
Joe Conway AustinPUG
OverviewMethodology
CodeTest Cases
Results
PreliminaryFinalFuture
Raw Results
select sql || $$:$$ || numiter || $$:$$ || loc || $$:$$ ||
comm || $$:$$ || connstrat || $$:$$ || prnt || $$:$$ || dest
AS testcase
,iface || $$:$$ || extra as candidate
,round(duration::numeric, 3) as duration
,round((((duration - first_value(duration) OVER w)/
(first_value(duration) OVER w)) * 100)::numeric, 2)
AS pct_slower
,rank() OVER w
FROM
(
select iface,loc,comm,connstrat,sql,numiter,extra,prnt,dest
,avg(duration) as duration
from vtestresults group by 1,2,3,4,5,6,7,8,9
) as t
WINDOW w
AS (PARTITION BY sql,numiter,loc,comm,connstrat,prnt,dest
ORDER BY duration)
;
Joe Conway AustinPUG
OverviewMethodology
CodeTest Cases
Results
PreliminaryFinalFuture
sql001:50k:lcl:ipc:0:p-all:file
sql001:50k ⇒ Query 1 row, 50k times
lcl:ipc ⇒ Local host via IPC connection
0 ⇒ Connect every query
p-all:file ⇒ print every row to file
candidate | duration
-------------+----------
libpq:acc-n | 162.720
libpq:acc-a | 163.026
libpq:acc-y | 164.761
ODBC:acc-n | 236.307
Joe Conway AustinPUG
OverviewMethodology
CodeTest Cases
Results
PreliminaryFinalFuture
sql001:50k:lcl:ipc:1:p-all:file
sql001:50k ⇒ Query 1 row, 50k times
lcl:ipc ⇒ Local host via IPC connection
1 ⇒ Connect once
p-all:file ⇒ print every row to file
candidate | duration
-------------+----------
libpq:acc-n | 7.753
libpq:acc-a | 7.978
libpq:acc-y | 8.619
ODBC:acc-n | 14.321
Joe Conway AustinPUG
OverviewMethodology
CodeTest Cases
Results
PreliminaryFinalFuture
sql001:50k:lcl:tcp:0:p-all:file
sql001:50k ⇒ Query 1 row, 50k times
lcl:tcp ⇒ Local host via TCP/IP connection
0 ⇒ Connect every query
p-all:file ⇒ print every row to file
candidate | duration
-------------+----------
libpq:acc-a | 180.219
libpq:acc-n | 180.240
libpq:acc-y | 182.211
JDBC:acc-y | 195.415
JDBC:acc-n | 201.378
ODBC:acc-n | 253.118
Joe Conway AustinPUG
OverviewMethodology
CodeTest Cases
Results
PreliminaryFinalFuture
sql001:50k:lcl:tcp:1:p-all:file
sql001:50k ⇒ Query 1 row, 50k times
lcl:tcp ⇒ Local host via TCP/IP connection
1 ⇒ Connect once
p-all:file ⇒ print every row to file
candidate | duration
-------------+----------
libpq:acc-n | 9.733
libpq:acc-a | 9.815
libpq:acc-y | 10.628
JDBC:acc-y | 12.673
ODBC:acc-n | 19.645
JDBC:acc-n | 20.067
Joe Conway AustinPUG
OverviewMethodology
CodeTest Cases
Results
PreliminaryFinalFuture
sql001:50k:rmt:tcp:0:p-all:file
sql001:50k ⇒ Query 1 row, 50k times
rmt:tcp ⇒ Remote host via TCP/IP connection
0 ⇒ Connect every query
p-all:file ⇒ print every row to file
candidate | duration
-------------+----------
libpq:acc-n | 412.389
libpq:acc-a | 413.264
libpq:acc-y | 428.484
JDBC:acc-y | 449.693
JDBC:acc-n | 452.799
ODBC:acc-n | 536.782
Joe Conway AustinPUG
OverviewMethodology
CodeTest Cases
Results
PreliminaryFinalFuture
sql001:50k:rmt:tcp:1:p-all:file
sql001:50k ⇒ Query 1 row, 50k times
rmt:tcp ⇒ Remote host via TCP/IP connection
1 ⇒ Connect once
p-all:file ⇒ print every row to file
candidate | duration
-------------+----------
libpq:acc-n | 26.457
libpq:acc-a | 26.483
libpq:acc-y | 41.876
JDBC:acc-y | 44.093
JDBC:acc-n | 52.347
ODBC:acc-n | 53.972
Joe Conway AustinPUG
OverviewMethodology
CodeTest Cases
Results
PreliminaryFinalFuture
sql50k:001:rmt:tcp:1:p-all:file
sql50k:001 ⇒ Query 50k rows, 1 time
rmt:tcp ⇒ Remote host via TCP/IP connection
1 ⇒ Connect once
p-all:file ⇒ print every row to file
candidate | duration
-------------+----------
libpq:acc-a | 0.621
libpq:acc-y | 0.665
JDBC:acc-y | 0.674
libpq:acc-n | 0.736
ODBC:acc-n | 1.075
JDBC:acc-n | 2.611
Joe Conway AustinPUG
OverviewMethodology
CodeTest Cases
Results
PreliminaryFinalFuture
sql10m:001:rmt:tcp:1:p-all:file
sql10m:001 ⇒ Query 10m rows, 1 time
rmt:tcp ⇒ Remote host via TCP/IP connection
1 ⇒ Connect once
p-all:file ⇒ print every row to file
candidate | duration
-------------+----------
libpq:acc-a | 37.768
libpq:acc-y | 40.196
JDBC:acc-y | 41.706
libpq:acc-n | 55.658
ODBC:acc-n | 82.782
JDBC:acc-n | 424.858
Joe Conway AustinPUG
OverviewMethodology
CodeTest Cases
Results
PreliminaryFinalFuture
sql10m:001:lcl:tcp:1:p-all:file
sql10m:001 ⇒ Query 10m rows, 1 time
lcl:tcp ⇒ Local host via TCP/IP connection
1 ⇒ Connect once
p-all:file ⇒ print every row to file
candidate | duration
-------------+----------
libpq:acc-y | 27.021
libpq:acc-a | 28.960
JDBC:acc-y | 31.661
libpq:acc-n | 37.058
ODBC:acc-n | 65.726
JDBC:acc-n | 407.906
Joe Conway AustinPUG
OverviewMethodology
CodeTest Cases
Results
PreliminaryFinalFuture
sql10m:001:lcl:tcp:1:p-all:file
sql10m:001 ⇒ Query 10m rows, 1 time
lcl:ipc ⇒ Local host via IPC connection
1 ⇒ Connect once
p-all:file ⇒ print every row to file
candidate | duration
-------------+----------
libpq:acc-y | 26.744
libpq:acc-a | 28.902
libpq:acc-n | 33.926
ODBC:acc-n | 63.34
Joe Conway AustinPUG
OverviewMethodology
CodeTest Cases
Results
PreliminaryFinalFuture
Final Results
To be continued . . .
Joe Conway AustinPUG
OverviewMethodology
CodeTest Cases
Results
PreliminaryFinalFuture
Future Work
Additional interfaces
Different versions of JDBC libraryAlternate JDBC librarynpgsql (.Net data provider)py-postgresql (native python interface)
Additional alternate fetch strategies
Prepared statementsServer-side COPY with client-side file fetch
Cost of SSL connection
Joe Conway AustinPUG
OverviewMethodology
CodeTest Cases
Results
PreliminaryFinalFuture
Questions?
Thank [email protected]
Joe Conway AustinPUG