Home > Technology > Writing infinite scalability web applications with PHP and PostgreSQL

Writing infinite scalability web applications with PHP and PostgreSQL

Date post: 06-May-2015
Author: gabriele-bartolini
View: 19,008 times
Download: 1 times
Share this document with a friend
PostgreSQL 9.2 introduced native support for the JSON data type, as well as V8/Javascript and Coffeescript procedural languages. Learn how you can write web applications in PHP using an intelligent and horizontally sharded cluster of PostgreSQL databases, bringing you infinite scalability and parallel processing. This talk will guide you through the development lifecycle of the application, focusing on architecture, technologies, testing and deployment.
Embed Size (px)
of 70 /70
Writing infinite scalability web applications with PHP and PostgreSQL PHPDay 2013 - Verona Gabriele Bartolini 2ndQuadrant Italia [email protected] @_GBartolini_ sabato 18 maggio 13
  • 1.Writing innite scalability webapplications with PHP andPostgreSQLPHPDay 2013 - VeronaGabriele Bartolini2ndQuadrant [email protected]@_GBartolini_sabato 18 maggio 13

2. Gabriele BartoliniCo-Founder and Manager of 2ndQuadrant Italia Data Architect, Business critical environments (OLTP) Data warehousingCo-Founder Italian PostgreSQL Users GroupCo-Founder PostgreSQL EuropePostgreSQL Contributor and Advocatesabato 18 maggio 13 3. Gabriele Bartolini 1.0 (1995-2005)Web miningWeb programmer C, C++ and PHP Web accessibility IWA W3C (EARL)Open source developer ht://Dig (C++) ht://Check (C++, MySQL,PHP) ht://Miner (C++,PostgreSQL)sabato 18 maggio 13 4. PostgreSQL = Postgres100% open source, available at www.postgresql.orgsabato 18 maggio 13 5. Whos using Postgres?... in a production environmentsabato 18 maggio 13 6. Whos planning to use Postgres?sabato 18 maggio 13 7. Whos familiar with sharding?sabato 18 maggio 13 8. Traditional PHP applicationHTML 5JavascriptCSS 3PHPPostgresJSONResultSetsabato 18 maggio 13 9. Postgres 9.2+ PHP ApplicationHTML 5JavascriptCSS 3PHPPostgresJSONsabato 18 maggio 13 10. HTML 5JavascriptCSS 3PHPPostgresPL/ProxyPostgres PostgresPostgresNoSQL......NoSQL with Postgressabato 18 maggio 13 11. ShardingHorizontal partitioningtechniqueLogically and physicallysplit a table by distributingdata across differentdatabases/servers Reduced number of rows ineach shard Reduced index sizeSome DBMS have the"DISTRIBUTED BY" clausefor "CREATE TABLE"Query distribution Parallel processing Shared nothing architecturesabato 18 maggio 13 12. Table 1Table 1Shard 1Table 1Shard 2Table 1Shard 3Table 1Shard ...Table 1Shard nTable 1SELECT * FROM Table1;Parallel execution on the n shardssabato 18 maggio 13 13. Part IPostgre(s|SQL)sabato 18 maggio 13 14. PostgreSQLStarted in 1995 (Postgres started in 1986)Current major version is 9.2 Supported major versions are 8.4, 9.0, 9.1 and 9.2The PostgreSQL License (BSD style) Not subject to monopoly nor acquisitionsPlatform independent (C language)sabato 18 maggio 13 15. Some major featuresClient/ServerConcurrency (MVCC)Referential integrityACID transactionsViewsFunctionsTriggersTablespacesPartitioningSchemasPoint In Time RecoveryMaster/Slave Replicationsabato 18 maggio 13 16. Features for developersStandard compliant (e.g.SQL, XML, UTF8, JSON, ...)Client interfaces: C/C++,PHP, JDBC, ODBC, .Net,Python, Perl, Ruby, ...Vast gamma of ORMsSeveral procedurallanguagesVersatile and extensible custom data types intelligence in the database extensions (from 9.1) PostGISDatabase Unit testingsabato 18 maggio 13 17. PostgreSQL 9.3Autumn 2013JSON operators andfunctionsUpdatable viewsMaterialised viewsWritable foreign tablesSupport for LATERAL...http://wiki.postgresql.org/wiki/What%27s_new_in_PostgreSQL_9.3sabato 18 maggio 13 18. NO ... DISTRIBUTED BYPostgreSQL does not support transparent distribution ofdata and queries (yet)Sharding can be implemented through PL/Proxy usingfunctionssabato 18 maggio 13 19. Part IIArchitecture and scalabilitysabato 18 maggio 13 20. ScalabilityProperty of a database to grow according to bothtechnical and business requirementssabato 18 maggio 13 21. ServerdbdbSingleSMP ServerVertical scalabilitysabato 18 maggio 13 22. ServerdbCommodityHardwareServer ServerdbHorizontal scalabilitysabato 18 maggio 13 23. DisclaimerFor the sake of simplicity, every database serverdescribed in the following architecture diagrams,deliberately represents a single point of failure.Due to its high availability/disaster recovery features andopen source tools, PostgreSQL is widely used in missioncritical environments.sabato 18 maggio 13 24. ServerpostgresPhysical/VirtualserverSingle server architectureapache/phpPros: costCons: performance (CPU, RAM, I/O), scalability, businesscontinuity, security, no db connection poolingsabato 18 maggio 13 25. Web serverTwo-server architectureapache/phpPros: exibility (hardware, OS), cost optimisation, security,performance (CPU, RAM, I/O), db connection poolingCons: business continuity, scalabilityDB ServerpostgrespgbouncerVirtual, +CPU, +RAM, -I/OPhysical, =CPU, =RAM, +I/Osabato 18 maggio 13 26. Benchmark2 x Intel Xeon E5630 CPUs 48 GB RAM Storage: 2 SAS 7.2k rpm, RAID 1 2 SAS 15k rpm, RAID 1 6 SAS 15k rpm, RAID 1+0850 MB/s seq read13k UPDATE/sec 32 clients 1.1k/sec for large dbsPostgreSQL 9.2/CentOS 6Rental costs: 1.8k setup + 350 /monthsabato 18 maggio 13 27. Web serverMultiple web servers architectureapache/phpDB ServerpostgrespgbouncerpgbouncerWeb serverapache/phppgbouncerHA ProxyHTTP BalancingPros: scalability andcontinuity (Web)Cons: scalabilityand continuity (DB)sabato 18 maggio 13 28. Web serverRead only database serversapache/phpDB MasterpostgrespgbouncerpgbouncerWeb serverapache/phppgbouncerHA ProxyHTTP BalancingDB Hot StandbypostgrespgbouncerStreamingReplicationRORORW RWsabato 18 maggio 13 29. Web serverDatabase sharding architectureDB Access ServerpostgrespgbouncerWeb serverHA ProxyWeb layerpl/proxyDatabase Access LayerDB Shard ...postgrespgbouncerDB Shard npostgrespgbouncerDB Shard 1postgrespgbouncerDatabaseData LayerNoSQLNoSQLInnite horizontal scalabilitySharednothingarchitecturesabato 18 maggio 13 30. Architecture recapDo not get carried away by current trends and buzzwordsMake your decisions about architecture based on: business continuity requirements results of both database and application benchmarksAlways start with a simple architecture (two-server)Enhance gradually up to sharding for innite scalabilitysabato 18 maggio 13 31. Part III(Open Source) Technologiessabato 18 maggio 13 32. PgBouncerLightweight connectionpooler for PostgreSQLWritten in C by SkypeRecycling connections on: sessions transactions statements (for PL/Proxy)Virtual databases Real databases can be ondifferent backend serversOnline reconguration,restart and upgrade Online statistics Supports PAUSE/RESUMEsabato 18 maggio 13 33. Conguration example[databases]db1_ro = host=pg2 dbname=db1 port=5432* = host=pg1 port=6432[pgbouncer]listen_addr = *listen_port = 6432pool_mode = session...sabato 18 maggio 13 34. StatementTransactionSession/Transaction/StatementConnection/SessionTransactionINSERTUPDATETransactionINSERTSELECTUPDATEBEGINCOMMITsabato 18 maggio 13 35. hstoreData type in PostgreSQLKey/value pairs as a singlevalueUseful for storing: unstructured data many attributes for arecordKeys and values are storedas textSet of operators ->, @>, ?, ...IndexesIntegrates with JSON in 9.3sabato 18 maggio 13 36. JSON SupportIntroduced in PostgreSQL 9.2Data typeGenerators (9.3)Operators (9.3)Integration with hstore (9.3)sabato 18 maggio 13 37. CREATE TABLE pictures (id SERIAL PRIMARY KEY,user_id INTEGER NOT NULLREFERENCES users(id),title TEXT,metadata JSON,thumbnail BYTEA,picture BYTEA,created_at TIMESTAMP DEFAULT now(),updated_at TIMESTAMP DEFAULT now());sabato 18 maggio 13 38. INSERT INTO pictures(user_id, title, metadata)VALUES (100,A picture of my Stratocaster,{"latitude":43.883,"longitude":11.1,"width":2240,"height":1680});Storing a JSON valuesabato 18 maggio 13 39. WITH m AS (SELECT 43.883 AS latitude,11.1 AS longitude,2240 AS width, 1680 AS height)INSERT INTO pictures(user_id, title, metadata)SELECT 100,Another picture of my Stratocaster,to_json(m) FROM m;Storing a JSON value #2sabato 18 maggio 13 40. WITH p AS (SELECT title, metadata, created_atFROM pictures WHERE id = 1)SELECT to_json(p) AS json FROM p;{"title":"A picture of my Stratocaster","metadata":{"latitude":43.883,"longitude":11.1,"width":2240,"height":1680},"created_at":"2013-05-1113:11:37.788636"}9.3 Returning a JSON objectsabato 18 maggio 13 41. WITH p AS (SELECT id, title, metadata, created_atFROM pictures WHERE user_id = 100)SELECT json_agg(p) AS json FROM p;[{"id":1,"title":"A picture of my Stratocaster","metadata":{"latitude":43.883,"longitude":11.1,"width":2240,"height":1680},"created_at":"2013-05-11 13:15:21.078747"}, +{"id":2,"title":"Another picture of my Stratocaster","metadata":{"latitude":43.883,"longitude":11.1,"width":2240,"height":1680},"created_at":"2013-05-11 13:15:21.078747"}]9.3 Returning an array of JSONssabato 18 maggio 13 42. 9.3 A taste of JSON operatorsSELECT id, (metadata->longitude,metadata->latitude) AS coordsFROM picturesWHERE user_id = 100;id | coords----+---------------1 | (11.1,43.883)2 | (11.1,43.883)...sabato 18 maggio 13 43. PL/JavascriptProcedural LanguageAllows to implement functions inside PostgreSQL using theJavascript LanguageAvailable as an extensionPowered by V8 JavaScript Enginesabato 18 maggio 13 44. CREATE FUNCTION plv8_test(keys TEXT[],vals TEXT[]) RETURNS TEXTLANGUAGE plv8 IMMUTABLE STRICT AS$BODY$var o = {};for(var i=0; i 1Queries are run in auto-commit mode PgBouncer (statement)sabato 18 maggio 13 46. The need for an APIPL/Proxy forces to workwith functionsOne database cluster with: access/proxy layer data layer (shards)Database cluster ~Singleton classBusiness logic in thedatabaseProgrammers need todene an API with access/authorisation rules Proxy functions (PL/Proxy) Remote functions (any)Supports SQL/MEDsabato 18 maggio 13 47. Database sharding architecturepg-access-1postgrespl/proxypg-data-2postgrespg-data-1postgresDatabaseAccessLayerDatabaseDataLayers00 s01 s02 s03- Denition of the cluster- Data/Queries distributed by user (hash)- Functions using the plproxy language- User data tables- Functions (any language)sabato 18 maggio 13 48. Dene the cluster with SQL/MEDCREATE SERVER myclusterFOREIGN DATA WRAPPER plproxyOPTIONS (connection_lifetime 1800,s0 dbname=shard00 host=pg-data-1,s1 dbname=shard01 host=pg-data-1,s2 dbname=shard02 host=pg-data-2,s3 dbname=shard03 host=pg-data-2);sabato 18 maggio 13 49. Proxy function for GETCREATE OR REPLACE FUNCTIONget_pictures(i_uid INTEGER)RETURNS jsonLANGUAGE plproxyAS$BODY$CLUSTER mycluster;RUN ON hashint4(i_uid);$BODY$;sabato 18 maggio 13 50. Data layer function for GETCREATE OR REPLACE FUNCTIONget_pictures(i_uid INTEGER)RETURNS jsonLANGUAGE SQL AS$BODY$WITH p AS (SELECT id, title, metadata, created_atFROM pictures WHERE user_id = $1) SELECT json_agg(p) AS json FROM p;$BODY$;sabato 18 maggio 13 51. Proxy function for INSERTCREATE OR REPLACE FUNCTIONinsert_picture(i_uid INTEGER, i_titleTEXT, i_metadata JSON, i_thumbnail BYTEA,i_picture BYTEA)RETURNS INTEGER LANGUAGE plproxyAS $BODY$CLUSTER mycluster;RUN ON hashint4(i_uid);$BODY$;sabato 18 maggio 13 52. Data function for INSERTCREATE OR REPLACE FUNCTIONinsert_picture(i_uid INTEGER, i_titleTEXT, i_metadata JSON, i_thumbnail BYTEA,i_picture BYTEA)RETURNS INTEGER LANGUAGE SQLAS $BODY$INSERT INTO pictures (user_id, title,metadata, thumbnail, picture) VALUES ($1,$2, $3, $4, $5) RETURNING id;$BODY$;sabato 18 maggio 13 53. Data representation in PHP// Obviously all of this will come from a form$user_id = 100;$title = A picture of my Stratocaster;$metadata = array (latitude => 43.883,longitude => 11.1,width => 2240,height => 1680, // ... EXIF?);$thumbnail = ... // Binary stream$picture = ... // Binary streamsabato 18 maggio 13 54. INSERT using PDO// $access_layer_conninfo = pgsql:...$connection = new PDO($access_layer_conninfo);$stmt = $connection->prepare(SELECTinsert_picture(?, ?, ?, ?, ?)); // NOSQL!$stmt->bindParam(1, $user_id, PDO::PARAM_INT);$stmt->bindParam(2, $title, PDO::PARAM_STR);$stmt->bindParam(3, json_encode($metadata),PDO::PARAM_STR);$stmt->bindParam(4, $thumbnail, PDO::PARAM_LOB);$stmt->bindParam(5, $picture, PDO::PARAM_LOB);$stmt->execute(); // TODO: Check and get the IDsabato 18 maggio 13 55. MemcachedDistributed system for caching of objects in memory key/value pairs volatile datapgmemcache extensionIntegration of database and applications (read scaling)sabato 18 maggio 13 56. PostgreSQLMemcachedm[k] = vPHPtrigger+pgmemcachesabato 18 maggio 13 57. Part IVTesting and deploymentsabato 18 maggio 13 58. Test-Driven DB developmentDatabase developmenttechniqueWrite tests before actualfeaturespgTAP is an extension forTDDD in PostgreSQLAllows to check for objects,privileges, functions,triggers, data, queries, etc.TransactionalProvides with statistics via SQL or pg_proveSupports Test AnythingProtocol (TAP) Continuous integrationsabato 18 maggio 13 59. BEGIN;SELECT plan(3);-- insert test 1 here-- insert test 2 here-- insert test 3 hereSELECT * FROM finish();ROLLBACK;pgTAP group test examplesabato 18 maggio 13 60. SELECT results_eq(SELECT * FROM active_users(),SELECT * FROM users WHERE active,active_users() should return activeusers);pgTAP example with results_eqsabato 18 maggio 13 61. Deployment with EXTENSIONExtensibilityDeployment is a crucial partof any ICT organisationPostgreSQL 9.1 introducesEXTENSION for packagingin-database applicationsFully transactional (atomic)Allows for upgrades anddowngradesSimplies logical dump/restore of objectsDevOps with EXTENSION: Devs write extensions Ops deploy themsabato 18 maggio 13 62. Extensions in detailExtensions require developer to write: Control le (metadata, settings, dependencies, ...) Installation scripts (SQL statements) Update scripts (SQL statements, optional) Source code / binaries (optional)sabato 18 maggio 13 63. Extension management-- INSTALLATIONCREATE EXTENSION pictures_app;-- REMOVALDROP EXTENSION pictures_app;-- UPDATE (fully atomic)ALTER EXTENSION pictures_app UPDATE;sabato 18 maggio 13 64. Part VConclusionssabato 18 maggio 13 65. ConclusionsPostgres is much more than a traditional databasePHP integrates perfectly with it, in multiple avours (nativeinterface, PDO, Doctrine, etc.)Start exploiting vertical scalabilityAlways perform benchmark!Evaluate costs and complexity of horizontal scalabilityPostgres is business friendly in terms of licensingsabato 18 maggio 13 66. Feel free to approach me aboutOpen Source CommunityLicenseTotal Cost of OwnershipDevelopment life cycleQuality of code anddocumentationBusiness ContinuitySecurity and data protectionPostgreSQL Eco-systemPerformance and scalabilitySoftware developmentsabato 18 maggio 13 67. CommunityNext events: PGDay.IT (Autumn 2013) PGConf.EU (Dublin, Oct 29- Nov 1, 2013)Italian PostgreSQL UsersGroup (ITPUG) www.itpug.orgPostgreSQL Europe www.postgresql.eusabato 18 maggio 13 68. ReferencesPostgreSQL: www.postgresql.orgPgBouncer: http://wiki.postgresql.org/wiki/PgBouncerPL/Proxy: http://wiki.postgresql.org/wiki/PL/ProxyPL/V8: http://pgxn.org/dist/plv8/pgTAP: www.pgtap.orgsabato 18 maggio 13 69. Questions?E-mail: [email protected]: @_GBartolini_sabato 18 maggio 13 70. Thank you!Copyright (c) 2013 2ndQuadrant ItaliaLicense: Creative Commons BY-NC-SA 3.0http://creativecommons.org/licenses/by-nc-sa/3.0/it/deed.ensabato 18 maggio 13