© 2015 EnterpriseDB Corporation. All rights reserved. 1
Introducing Postgres Plus Advanced Server 9.4 & Postgres Enterprise Manager 5.0
© 2015 EnterpriseDB Corporation. All rights reserved. 2
• EnterpriseDB Overview • Postgres Plus Advanced Server 9.4
− Resource Management − Partitioning − SQL/Protect & MTK Enhancements
• Highlights of What’s New in PEM 5.0 − Enhanced Alerting − Log Analysis Expert − Custom Probes − Auto-Discovery of Managed Servers − Remote Monitoring
Agenda
© 2015 EnterpriseDB Corporation. All rights reserved. 3
POSTGRES innovation
ENTERPRISE reliability
24/7 support
Services & training
Enterprise-class features, tools &
compatibility
Indemnification
Product road-map
Control
Thousands of developers
Fast development
cycles
Low cost
No vendor lock-in
Advanced features
Enabling commercial adoption of Postgres
© 2015 EnterpriseDB Corporation. All rights reserved. 4
EDB is a Gartner Magic Quadrant Leader
© 2015 EnterpriseDB Corporation. All rights reserved. 5
• EnterpriseDB Overview • Postgres Plus Advanced Server 9.4
− Resource Management − Partitioning − SQL/Protect & MTK Enhancements
• Highlights of What’s New in PEM 5.0 − Enhanced Alerting − Log Analysis Expert − Custom Probes − Auto-Discovery of Managed Servers − Remote Monitoring
Agenda
© 2015 EnterpriseDB Corporation. All rights reserved. 6
from PostgreSQL core from EDB Development
• 64 bit LOBs up to 4TB in size
• Custom background workers
• Writable Foreign Data Wrappers
v9.1
EDB contributions to PostgreSQL core
• No restore In-place version upgrades
v9.2
v9.3
v9.0
• Materialized Views
• Deferrable unique constraints and Exclusion constraints
• Streaming replication
• Windows 64 bit Support
• Hot standby
• Synchronous replication
• Serializable Snapshot Isolation
• In-memory (unlogged) tables
• Writeable Common Table Expressions (WITH)
• Cascaded streaming replication
• JSON support, Range Types
• VARRAY support • SQL Profiler
• Index Advisor • Parallel Bulk Data
Load
• Row Level Security • Declarative Partitioning syntax
• Table() function support for nested tables
• INSERT APPEND hint
• xDB Multi-master replication
• Expanded Object Type support
• Partition Read Improvements over 75x
• Support for 1000s of Partitions
• Partition write improvements over 400x
• MySQL Foreign Data Wrappers for SQL/MED
Postgres Plus Advanced Server Key Feature Development
• Index-only scans (covering indexes)
• Linear read scalability to 64 cores
v9.4 • pg_prewarm • ALTER SYSTEM • Concurrently updatable Materialized Views
• Mongo FDW & MySQL FDW
• Logical Decoding for Scalability
• JSONB Data Type
• JSONB
Indexing • Expanded
JSON functions • Delayed
Application of Replication
• 3x Faster GIN indexes
• Support for Linux Huge Pages
• CPU & I/O Resource Management
• SQL Aggregation with CUBE, ROLLUP and GROUPING SETS
• Comprehensive UTL_HTTP Package
• Hash Partitioned Tables
• Connect_By_Root Operator for hierarchical
queries • SQL/Protect
Logging to DB Table
• EDB*Loader Improved Error handling
© 2015 EnterpriseDB Corporation. All rights reserved. 7
CPU & I/O Resource Management
Hash Partitioned Tables
SQL Aggregation with CUBE, ROLLUP and GROUPING SETS
Comprehensive UTL_HTTP Package
Connect_By_Root Operator
ICU Collation
EDB*Loader Improvements
SQL/Protect Logging to DB Table
Migration Toolkit Enhancements
Postgres Plus Advanced Server Postgres Community
Feature Highlights for Postgres Plus Advanced Server (PPAS) 9.4
Many new features including: Logical Change Set Extraction JSONB Data Type Time Delayed Standby ALTER SYSTEM pg_prewarm() Materialized View Refresh Concurrently Ordered Set Aggregates and more… http://www.depesz.com/ is a good reference site https://commitfest.postgresql.org/ is the global community site for patches review
© 2015 EnterpriseDB Corporation. All rights reserved. 8
Postgres Plus Advanced
Server
Resource Manager
(CPU & I/O)
Reporting
Transactions
80%
20%
Run Mixed Workloads More Efficiently with PPAS 9.4 Resource Management
• DBA assigns CPU & I/O to job groups
• Allocates and prioritizes consumption of resources
• Low priority jobs don’t hurt high priority jobs
© 2015 EnterpriseDB Corporation. All rights reserved. 9
• Create Resource Groups and assign the CPU Rate Limit
• Use these resource groups during a psql session
Run Mixed Workloads More Efficiently with PPAS 9.4 Resource Management
- Statements executed will be limited in CPU or writing to shared_buffers per resource group. - Individual limits are computed based on recent CPU / shared_buffer usage. - Processes sleep when necessary & avoid sleep when holding critical locks. - The limits are adjusted regularly based on current usage. - If multiple processes in the same group are being executed, aggregate usage will be limited
CREATE RESOURCE GROUP resgrp_a; CREATE RESOURCE GROUP resgrp_b; ALTER RESOURCE GROUP resgrp_a SET cpu_rate_limit = .25; ALTER RESOURCE GROUP resgrp_a SET dirty_rate_limit = 12288;
SET edb_resource_group TO res_grp_a;
© 2015 EnterpriseDB Corporation. All rights reserved. 10
Chapter 13 of EDB Oracle Compatibility Guide for details on usage
One logically large table is broken into smaller physical pieces. • Worthwhile when a table would otherwise be very large. • Exact point to see benefits will depend on the application.
When should I Partition? • Good rule of thumb is that the size of the table should exceed the physical memory
of the database server. • When most accessed rows are in a single partition or a small number of partitions. • When there is a lot of concurrent inserts or updates (Hash partitioning may benefit) • When a query or update accesses a large percentage of a single partition. • For Bulk Loads / Unloads (ALTER TABLE faster than bulk load, avoids VACUUM
overhead from DELETE). • When actively archiving seldom-used data to less expensive storage.
Support Larger Tables with Partitioning
© 2015 EnterpriseDB Corporation. All rights reserved. 11
Use PPAS Syntax To Minimize Partitioning Errors and Complexity
CREATE TABLE sales ( dept_no number, part_no varchar2, country varchar2(20), date date, amount number ) PARTITION BY RANGE(date) ( PARTITION q1_2014 VALUES LESS THAN('2014-Apr-01'), PARTITION q2_2014 VALUES LESS THAN('2014-Jul-01'), PARTITION q3_2014 VALUES LESS THAN('2014-Oct-01'), PARTITION q4_2014 VALUES LESS THAN('2015-Jan-01') ); CREATE INDEX sales_date on sales(date);
Simple Declarative PARTITION BY and SUBPARTITION BY
Single Index command
• More SQL syntax provide sophisticated data management techniques − ADD / DROP to augment the partitions − SPLIT to divide the data − EXCHANGE to swap in a new partition − TRUNCATE to remove all data but leave
structure in tact − MOVE to shift data to a different tablespace
• PPAS Improved performance with Fast Pruning and Constraint Exclusion support
• System Catalog Views for Partitions − ALL_PART_TABLES − ALL_TAB_PARTITIONS,
ALL_TAB_SUBPARTITIONS − ALL_PART_KEY_COLUMNS,
ALL_SUBPART_KEY_COLUMNS
© 2015 EnterpriseDB Corporation. All rights reserved. 12
List, Range or Hash Partition Rules • Provide the constraints to define where data is stored • For PPAS, also used to support Fast Partition Pruning • Consider how data stored will be queried, include often-queried columns
in partitioning rules.
• List – Single partitioning key column; based on exact value • Range – One of more partitioning key columns; based on values between
two extremes • Hash (New 9.4) – Data divided amongst equal sized partitions based on
hash value. * Internal tests have shown hash partitioning can improve performance when many hundred concurrent connections insert/update to the same table*
PPAS 9.4 Supports Various Partitioning Rules
© 2015 EnterpriseDB Corporation. All rights reserved. 13
Advanced Server’s Query Planner uses two optimization techniques to compute an efficient plan: • Constraint exclusion (constraint_exclusion = partition or on)
− Provided by PSQL − SELECT w/ WHERE: Query Planner must examine the CHECK constraints defined
for each partition before deciding which partition to send query fragments to. • Fast pruning (edb_partition_pruning = on)
− Occurs earlier in query planner process. Understands the relationship between partitions in an Oracle style partitioned table.
− SELECT w/ WHERE: Query Planner can reason that only a certain partition holds the values without examining the constraints defined for each partition.
− Can be used for LIST or single value RANGE Partitions (not usable on subpartitioned tables or multi-value range partitioned tables)
− Works with >, >=, =, <=, <, AND, BETWEEN operators in the WHERE Clause
PPAS Improves Partitioning Performance
© 2015 EnterpriseDB Corporation. All rights reserved. 14
• SQL/Protect Logging to DB Tables (Ch 4.1.1.2.3 of EDB Enterprise Edition Guide) − Gives DBA a tool to protect against SQL Injection attacks − View edb_sql_protect_queries contains logged information − To test, enable SQL/Protect, create a test role, set blocking of unbounded
DML (for example), then execute UPDATE without WHERE clause on test table
• MTK Improvements − Migration Toolkit will now provide a detailed log with well defined error
codes to allow DBAs to better understand which capabilities of their database applications from Oracle, MySQL, SQL Server or Sybase are migrate-able to PPAS.
− See Ch 9 of Migration Toolkit documentation for a list of the error codes provided
More Flexibility with SQL/Protect and Migration Toolkit Enhancements
© 2015 EnterpriseDB Corporation. All rights reserved. 15
1. Customers running mixed workloads.
2. Application Developers who require communication with external Web servers.
3. Customers with large tables where they often search for exact matches or have many concurrent inserts/updates.
4. Users who think they need a NoSQL database.
5. Customers with reporting or data warehousing databases.
6. DBAs who use need to bulk load data.
7. DBA’s concerned with Security and SQL Injection Attacks.
Postgres Plus Advanced Server (PPAS) 9.4 Use Cases
© 2015 EnterpriseDB Corporation. All rights reserved. 16
• EnterpriseDB Overview • Postgres Plus Advanced Server 9.4
− Resource Management − Partitioning − SQL/Protect & MTK Enhancements
• Highlights of What’s New in PEM 5.0 − Enhanced Alerting − Log Analysis Expert − Custom Probes − Auto-Discovery of Managed Servers − Remote Monitoring
Agenda
© 2015 EnterpriseDB Corporation. All rights reserved. 17
Over 225 predefined and custom probes to support alerts via SMTP or SNMP
Predefined & customized at-a-glance dashboards
OS and database statistics collection
Replication monitoring
MONITOR TUNE MANAGE
CRUD operations on all database objects
Bulk operations across multiple servers
Capacity Manager to plan & forecast
Log Manager and Audit Manager to configure database metric collection
SQL/Profiler to speed up large workloads
Index Advisor to suggest and create indexes
Postgres Expert for best practice enforcement
Tuning Wizard for machine utilization and load profiles
No Other Tool Provides Much Visibility Into your Postgres Databases
© 2015 EnterpriseDB Corporation. All rights reserved. 18
Monitor All Your Postgres Databases From One Screen
• Customized global dashboard
• View up/down status of all agents
• Monitor alerts from many servers in one place
• Navigate to Dashboards for further analysis
© 2015 EnterpriseDB Corporation. All rights reserved. 19
Highlights for PEM 5.0
• Alert Level Specific Notifications
• Log Analysis Expert
• Custom Probes
• Auto-Discovery of Managed Servers
• Remote Monitoring
• Download Tuning Recommendations
• Log OS Metrics for Backend Server Processes
• Better Zooming and Granularity Control
• Cross Hierarchy Charts
• Monitor Streaming Replication
Ease Of Use Richer Dashboards
© 2015 EnterpriseDB Corporation. All rights reserved. 20
Model your Support Staff Rotation in PEM for Email Alerting
• Decide who is externally notified of alerts by creating the email groups.
• Define different email envelope information, depending on time of day.
© 2015 EnterpriseDB Corporation. All rights reserved. 21
Automate Notification or Actions on over 225 Predefined or Customizable Alerts (1 of 2)
• Create and manage alerts
• Examples – running low on disk space, server down, last vacuum, total table bloat, etc.
• Set the low, medium and high threshold
© 2015 EnterpriseDB Corporation. All rights reserved. 22
Automate Notification or Actions on over 225 Predefined or Customizable Alerts (2 of 2)
• Choose when and who to send email notifications to.
• Decide when to send SNMP traps
• Further execute any external script.
© 2015 EnterpriseDB Corporation. All rights reserved. 23
• Generate reports on stats like locks and queries that are collected by the Log Manager
• Collects information based on historical csv data, not dependent on live logs.
• First configure Log Manager to enable collection of log files.
• Once you have logs collected, use the Log Analysis Expert to select which reports to generate
• More details in the ‘Log Analysis Expert’ section of the Online Help
Better Understand Database Activities with Log Analysis Expert
© 2015 EnterpriseDB Corporation. All rights reserved. 24
Gather Any Custom Information You Want With Custom Probes (1 of 3)
• Probes are used to gather metrics for alerts and dashboard charts.
• View all System Probes.
• Add New or Modify existing probes to suit your needs.
© 2015 EnterpriseDB Corporation. All rights reserved. 25
Gather Any Custom Information You Want With Custom Probes (2 of 3)
• Define the columns that will be used to store the data collected.
• If ‘Graphable’, the column will be available on Charts and Capacity Manager.
• Control if the metric is Point In Time or Cumulative
© 2015 EnterpriseDB Corporation. All rights reserved. 26
Gather Any Custom Information You Want With Custom Probes (3 of 3)
• If this is a SQL Probe, enter the SQL SELECT statement to be executed by the probe in the Code tab.
• If this is a Batch probe, enter the shell or .bat script to be invoked by the probe.
© 2015 EnterpriseDB Corporation. All rights reserved. 27
• Locate databases that reside on servers with Agents installed on them.
• After you install agents − Select them in the tree and
choose Management -> Auto Discovery to open the dialog shown.
− Then select the database server to have the server property fields filled in
• More details in the ‘Automatic Discovery of Server’ section of the Online Help
Auto Discovery of Managed Servers For Easier Deployment
© 2015 EnterpriseDB Corporation. All rights reserved. 28
• Allow remote monitoring of servers (without installing agents using direct JDBC connections), from the agent running on the PEM server − Ignores OS level stats,
disables features like Server Startup, Audit/Log/Capacity Manager, Tuning/Deployment Wizard
• Create Server and select Remote monitoring in PEM Agent Tab.
Remote Monitoring
© 2015 EnterpriseDB Corporation. All rights reserved. 29
Feature Benefit Audience Motivation
Alert Controls Control who gets which alerts depending on severity and time of day / day of week.
DBAs Ease of Use
Log Analysis Expert Better understanding the operations that occur in your database.
DBAs Ease of Use
Custom Probes Get whatever custom information you want from your databases or servers; pull in BART or EFM status information
DBAs Compatibility
Auto-discovery of Managed Servers
Easily configure PEM Server with all database servers on managed hosts.
DBAs Ease of Use
Remote Monitoring Monitor servers without installing agents using direct connections.
DBAs Ease of Use
Log OS Metrics for Backend Server Processes
New probe that capture memory and CPU alongside process information
DBAs Compatibility
Improved Dashboards (Better zooming / granularity control, cross server charts)
Compare metrics, More accurate and relevant information when zooming in and out of charts
DBAs Ease of Use
Recap: Main Feature/Benefits for this Release
© 2015 EnterpriseDB Corporation. All rights reserved. 30
How can I learn more?
• Download:
• http://www.enterprisedb.com/download-advanced-server • http://www.enterprisedb.com/download-postgres-enterprise-
manager
• General Information: • http://www.enterprisedb.com/postgres-plus-advanced-server • http://www.enterprisedb.com/postgres-enterprise-manager
© 2015 EnterpriseDB Corporation. All rights reserved. 31