Post on 14-Feb-2018
transcript
Database Platforms Best Practices and What’s New
Darrell Gosden
Principal Consultant
SAP Global Database and Technology Services
2 UKI SAP User Group 2013
Speaker Introduction Darrell Gosden – My Background
• Long term Sybase Principal Consultant (>20 Years) prior to SAP acquisition (in
Oct 2012 in the UK).
• Project / Customer Engagement Delivery – Financial Markets Bias.
• Relational Database / Data Management (including Oracle and Ingres) and
associated technologies.
• SAP Sybase Database Technology Specialist with core skill set in SAP Sybase
ASE, SAP Sybase IQ, SAP Sybase Replication Server and associated design /
operational management products & tools.
Contact Details: Darrell Gosden SCN Entry or you can always find me through LinkedIn!
Mobile: +44 (0) 7977257074
E-Mail: darrell.gosden@sap.com
Public SAP Community Blog(s) – or search “Darrell Gosden Running SAP Sybase Virtualized”:
Running SAP Sybase Data Management Products in a Virtualised World
The SAP Real-time Data Platform (RTDP)
“The Vision”
Database Platforms – Best Practices and What’s New Outline Agenda
Speaker Introduction Darrell Gosden – My Background
3 UKI SAP User Group 2013
Database Platforms – Best Practices and What’s New Outline Agenda
1. The SAP Real-time Data Platform (RDTP) “Vision” explained.
2. SAP Sybase Adaptive Server Enterprise (ASE) – What is it?
3. Migrating to SAP Sybase ASE.
4. ASE Customer Success Stories.
5. SAP Sybase IQ (IQ) – What is it / What’s New?
6. IQ Customer Success Stories.
7. The SAP Sybase IQ Near-line Storage (NLS) Solution – Overview.
8. SAP Sybase IQ NLS – The Rapid Deployment Service (RDS).
Please feel free to ask questions throughout the session.
Database Platforms – Best Practices and What’s New Outline Agenda
4 UKI SAP User Group 2013
The SAP Real-time Data Platform (RTDP)
“The Vision”
Database Platforms – Best Practices and What’s New Outline Agenda
The SAP Real-time Platform (RDTP) The “Vision”
5 UKI SAP User Group 2013
The SAP Real-Time Data Platform (RTDP) Where does SAP Sybase ASE fit in?
Database Platforms – Best Practices and What’s New Outline Agenda
The SAP Real-time Platform (RDTP) Where does SAP Sybase ASE fit in?
6 UKI SAP User Group 2013
SAP Sybase ASE Designed for mission-critical transactional environments
• Easily scale the amount
of users and data
• Execute millions of
transactions per minute
with terabyte-sized
databases
• Support rapid growth in
data volumes and
transaction levels
• Easy to maintain,
minimized downtime
• Automated tools for
monitoring and
diagnostics
• Reduce operational risk
by ensuring data security
and system stability
• Reliable and responsive
even under the most
punishing workloads
• More efficient use of
servers and storage
• Improve efficiency by
reducing administrative
tasks
Performance Efficiency Usability
Database Platforms – Best Practices and What’s New Outline Agenda
SAP Sybase ASE Designed for mission-critical transactional environments
7 UKI SAP User Group 2013
Sybase ASE – A Quick History
Certification with SAP Business Suite & NetWeaver
DBMS Clustering Database Clustering, Cluster Workload Mgmt,
Virtual Resource Management
Commodity & Low Cost Linux, Java-in-the-Database, XML
Heterogeneous Data Mgmt Heterogeneous Data Mgmt., Replication
Distributed Query Processing
Client/Server Client Server Architecture,
Stored Procedures, Triggers
1980S
1994: Microsoft releases SQL Server 4.2 for NT, MS & Sybase Split Marketing SQL Server
1990S 2009: Leader in Forrester Wave for Enterprise Database Management Systems
2000S
2010: “Best enterprise data management vendor on Wall Street four years running.”
2010S
IDC Finds ASE’s TCO to be 28% Less Than Other RDBMS Database Platforms – Best Practices and What’s New Outline Agenda
SAP Sybase ASE – A Record of Innovation
8 UKI SAP User Group 2013
IDC Finds ASE’s TCO to be 28% Less Than Other RDBMS
IDC research conducted in-depth interviews with organizations to develop insights into the true costs associated with deploying ASE in comparison with other RDBMS products.
Sybase ASE Five-Year Savings per 100 Users
IDC White Paper (Dec 2011): Calculating the True Cost of RDBMS Ownership and How Sybase ASE Stacks Up: A Guide for SAP Business Suite Users
“Sybase ASE installations were
more efficient than the other
systems, requiring 24% less
storage capacity and running on
29% less server processor
power.”
“Sybase ASE required
27% fewer IT staff resources
overall”
Database Platforms – Best Practices and What’s New Outline Agenda
IDC Finds SAP Sybase ASE’s TCO to be 28% Less Than Other RDBMS
9 UKI SAP User Group 2013
Migrating to SAP Sybase ASE Best Practices Thoughts – A (Very!) Quick Review
• Database Schema - Sybase PowerDesigner is your friend!
• Reverse Engineering.
• Schema Migration.
• Data Migration Approaches
• Big Bang Materialisation (Flat-file, ETL tool, Sybase CIS & “proxy tables”).
• Trickle Feed / Incremental (Flat-file, ETL tool, SAP Sybase Replication Server, Sybase CIS).
• Continuous Data Movement – SAP Sybase Replication Server.
• SQL / “code” Migration
• Hand craft / “home brew”.
• Tools provided as part of SAP Services packages – semi automated – targeted at Oracle to ASE.
• Parallel Running
• SAP Sybase Replication Server.
Database Platforms – Best Practices and What’s New Outline Agenda
Migrating to SAP Sybase ASE Best Practices – A (Very!) Quick Review
10 UKI SAP User Group 2013
Migrating to SAP Sybase ASE Best Practices – SAP Services Can Help You
A number of packaged SAP Services
offerings are available to support your
migration to SAP Sybase ASE
including:
• RDS Rapid Database Migration to SAP
Sybase ASE (inc. SAP applications).
• RDS Rapid Database Migration POC to
SAP Sybase ASE – to support a
modest proof-of-concept (PoC)
exercise. “Try before you buy!”.
• Database Migration of Custom
Environments to SAP Sybase ASE.
Please contact your SAP Account
Executive for fuller details – or –
myself during this conference if you
prefer.
Database Platforms – Best Practices and What’s New Outline Agenda
Migrating to SAP Sybase ASE Best Practices – SAP Services can help you!
11 UKI SAP User Group 2013
Migrating to SAP Sybase ASE Best Practices – Additional Migration Support
Go to the Public SAP Community Network (SCN) SAP Services Content Library
(http://scn.sap.com/docs/DOC-34352) where you will find an excellent series of
blogs by my college Jonathan Wesley-James on planning and executing SAP
Sybase ASE Migrations:
Database Platforms – Best Practices and What’s New Outline Agenda
Migrating to SAP Sybase ASE Best Practices – Useful Blogs
12 UKI SAP User Group 2013
Day & Zimmermann Engineering, Construction and Operations
“ As part of our technology transformation
initiative we migrated our SAP applications
running on the Oracle database to SAP
Sybase ASE. This project virtualized 10 SAP
applications consisting of 70+ servers with
VMware on Windows. Significant cost
savings were achieved on Sybase ASE
annual maintenance costs vs. Oracle, and,
using compression technology in SAP
Sybase ASE, our storage requirements
were reduced by 30% for additional
savings. SAP provided expertise and training
to ensure the migration went smoothly, and as
a satisfied long-time SAP customer we believe
SAP’s strategic roadmap for its data
management products will meet our needs
well into the future.
Trushar Desai, Director, SAP Technology,
Day & Zimmermann
”
Database Platforms – Best Practices and What’s New Outline Agenda
Day & Zimmermann Engineering, Construction and Operations
13 UKI SAP User Group 2013
Redington Gulf IT distribution and support services
Business Challenges
High Oracle database licensing and maintenance costs
Solution Implementation
Migrate SAP ERP system off Oracle to SAP Sybase ASE
Deploy new SAP CRM system on SAP Sybase ASE
SAP Services assisting with ERP migration
Benefits
100% performance improvement over Oracle
50% more compression than Oracle
DBA Cockpit provides simplified and better administration
Lower database licensing and maintenance costs
Lower database TCO
2x less expensive
than Oracle database
100% performance
improvement over
Oracle
Redington Gulf replaces Oracle database
with SAP Sybase ASE for lower TCO
IN BRIEF
Redington Gulf is the largest
an end-to end supply chain
provider for IT distribution and
support services across the
Middle East, Africa and
Turkey.
Database Platforms – Best Practices and What’s New Outline Agenda
Redington Gulf IT distribution and support services
14 UKI SAP User Group 2013
SAP Sybase IQ – Position within Real-time Data Platform
“Near Line Storage” (NLS)* & many other scenarios possible
* Discussed later.
IQ
16.0
Database Platforms – Best Practices and What’s New Outline Agenda
SAP Sybase IQ Real-time Data Platform (RTDP) Positioning
15 UKI SAP User Group 2013
SAP Sybase IQ Engine
Mu
ltiple
x g
rid a
rch
itectu
re
Storage area network
Loading engine
In-database analytics
Text search
Web-enabled analytics
Info
rmatio
n life
cycle
managem
ent
Low latency, write optimized store
Role-based access control
Communications and security
N-bit and tiered indexing
Column indexing
subsystem
Query engine
Column store
Aggressive scale out
Fully parallel
Re
silie
nt
We
b b
ase
d a
dm
inis
tratio
n a
nd
mo
nito
ring
LDAP authentication
Hash partitioned tables
and data affinity
New Generation
PETABYTE SCALE store
New indexing technology for increased compression and fast, incremental batch loads
SAP Sybase IQ In a Nutshell!
High performance, fully parallel bulk loading
Enterprise-grade RBAC authorization and LDAP authentication
Comprehensive web-based monitoring and administration
Low latency, concurrent write optimized delta store
Re-engineered column store for extreme data volumes
Resilient Multiplex grid withstands network interrupts and server failures
Aggressively scaled out query engine
Intelligent query engine with data affinity for “shared nothing” performance on a flexible “shared everything” architecture
Database Platforms – Best Practices and What’s New Outline Agenda
SAP Sybase IQ In a Nutshell!
16 UKI SAP User Group 2013
SAP Sybase IQ IQ Column Store and Compression – A cornerstone capability
IQ stores data vertically
Values of same column are stored next to each other, column values of a row are in
separate locations
Easy reference for fixed length values
Efficient evaluation for very large number of rows with a limited set of columns
Rows exist as row numbers which mark a position in each individual column vector
IQ compresses values
Dictionary Compression for repeating values, Storage Compression for all values
IQ is optimized for very large databases
IQ 16 holds the Guinness World Record for Big Data Load (> 34 TB / hr.)
Massive parallel Multicore and Multinode query execution, table partitioning
Database Platforms – Best Practices and What’s New Outline Agenda
SAP Sybase IQ IQ Column Store and Compression
17 UKI SAP User Group 2013
SAP Sybase IQ 16 What’s New?
Notable Enhancements in SAP Sybase IQ 16:
Improved table concurrency – multiple “writers” against a single table – “row level
locking/versioning” (RLV) feature NOTE: Part of REAL TIME option for IQ > uses memory based row versioning and row merge back techniques very similar
to SAP HANA. Some usage restrictions apply.
(Even!) better storage compression (NBit dictionary tokenization etc) – targeted at
VARCHAR/VARBINARY/other variable length data
(Even!) faster data loading – higher degrees of internal parallelism possible and data load
query plans available allowing for low level tuning/optimization Breaking News!: Early “like for like” IQ 15.4 v. IQ 16 comparisons indicate 6-8x faster loads with load
rates of 5,000,000 > 7,000,000 rows/sec easily achievable with IQ 16!
Query (inc. DQP) processing improvements – partition based hash algorithms & others
LDAP support.
Enhanced user security/authorisations based on fine grained access privileges/roles.
HADOOP Integration…HOT!
Database Platforms – Best Practices and What’s New Outline Agenda
SAP Sybase IQ 16 What’s New?
18 UKI SAP User Group 2013
SAP Sybase IQ 16 HADOOP Integration – Of particular note!
The data returned from a Hadoop analysis can be integrated into an SAP
Sybase IQ database in several ways:
• ETL Processing – Bulk load data from Hadoop data stores into SAP Sybase IQ using the
open source utility SCOOP.
• Data Federation – Expose HDFS files as tables in an SAP Sybase IQ database that
participate in SQL queries. The HDFS files do not need to be loaded into SAP Sybase IQ.
• Query Federation – Allow SQL queries in SAP Sybase IQ to execute Hadoop processes
that return data that is incorporated into the SQL result set.
• Client-side Federation – Federate queries across SAP Sybase IQ databases and Hadoop
files using the TOAD© SQL tool.
New capabilities such as these help position SAP Sybase IQ as part of the “SAP HANA
Platform for Big Data” – next slides!
Database Platforms – Best Practices and What’s New Outline Agenda
SAP Sybase IQ 16 HADOOP Integration – Of particular note!
19 UKI SAP User Group 2013
SAP
Enterprise
data
Non-SAP
Enterprise
data
Mobile data
Machine data (Sensors, SCADA, Machine
Logs, Etc.)
Data Sources
Analytics & Applications
HANA In Memory
Transactional Planning &
Simulation
Graph Analytical
Predictive
Analysis
Spatial
Extended
Storage (IQ)
NLS Solution
Tie
red S
tora
ge (H
ot-w
arm
-cold
)
Sm
art D
ata
Access
Dashboard / Reporting in Real-
Time
Large Low Cost Data Platform
(Hadoop / IQ)
Stream Processing
Real-Time
Replication
Synchronization
Historical Data, Offline Batch Processes,
Model Training etc.
SAP HANA Platform for Big Data
Use Cases •Energy Optimization
•Predictive
maintenance
•Remote asset mgmt.
•Supply/demand
forecast
•Inventory mgmt.
•Route optimization
SAP HANA Platform for Big Data Where does SAP Sybase IQ fit in?
Real-time data stream
(Billions of events/day)
Millions of events/day
correlated with Enterprise
Data
Enable real-time operations,
analysis and actions
Database Platforms – Best Practices and What’s New Outline Agenda
SAP HANA Platform for Big Data Where does SAP Sybase IQ fit in?
20 UKI SAP User Group 2013
SAP Sybase IQ In Summary
Ecosystem
Most mature
column store
Comprehensive
lifecycle tiering
MPP queries + virtual
marts + user scaling
High-speed
loads
Structured +
unstructured store
Comprehensive ANSI
SQL with OLAP
Built-in full-
text search
In-database analytics with
MapReduce + simulator
Web 2.0
APIs
Big data OpnSrc
APIs
Optimized BI and EIM;
model and replicate
Development and
administration tools Predictive analytics
Packaged ILM
applications
Bradmark, Symantec,
Whitesands, Quest,
and ZEND
SAS, SPSS, KXEN,
Fuzzy Logix,
Zementis, and Visual
Numerics
BMMSoft,
SOLIX, and PBS
SAP Sybase PowerDesigner,
SAP Sybase Replication
Server,
SAP, ISYS, and Panopticon
App. services
DBMS Hadoop
and R
Database Platforms – Best Practices and What’s New Outline Agenda
SAP Sybase IQ In Summary
21 UKI SAP User Group 2013
Financial Services Customers Succeed with IQ for
Mission-Critical Applications
Customer IQ Deployment
US Securities &
Exchange
Commission
With Sybase IQ, the SEC built a flexible, scalable and
reliable data-rich warehouse that delivers faster load
times and query results, and more comprehensive
information views—all at a lower cost.
Citigroup With Sybase powering their data archive, Citigroup
users can now research historical data themselves,
whereas in the past IT specialists were required to do
so.
Morgan Stanley
(Spain)
With Sybase IQ, Morgan Stanley Spain now has access
to heterogeneous system-stored critical data with single,
unified delivery that has dramatically reduced analysis
response times.
ICICI Bank ICICI bank has achieved improvement in system uptime
and query performance over its previous Teradata
implementation, in addition to the host of other benefits
of the Sybase IQ data warehouse migration.
BNP Paribas
Securities Services
Sybase IQ has improved the robustness of its online
reporting system. Now the bank can compile larger amounts
of data and anticipate growth in its customer base by
proactively providing real-time data.
Database Platforms – Best Practices and What’s New Outline Agenda
Customers Succeed with SAP Sybase IQ for Mission-Critical Applications
22 UKI SAP User Group 2013
SAP HANA and SAP Sybase ASE / IQ Working Together
Smart Data Access – A interesting new capability!
SAP HANA SP6 was released recently and
one of the key features included in this
release is the Smart Data Access, which
allows customers to combine SAP HANA
data with other heterogeneous data sources
like Hadoop, Teradata, SAP Sybase ASE
and SAP Sybase IQ.
With Smart Data Access (SDA) you can
expose data from remote sources as “virtual
tables” and combine these with other data in
HANA physical tables using HANA models.
There are few restrictions on usage of SDA
currently:
• Data in virtual tables is read only.
• Virtual tables do not support BLOB/CLOB “type”
data.
• Virtual tables cannot be used in multi node SAP
HANA clusters.
• Some analytical views do not support virtual
tables.
Database Platforms – Best Practices and What’s New Outline Agenda
SAP HANA and ASE / IQ Working Together Smart Data Access – A interesting new capability!
23 UKI SAP User Group 2013
Near-line Storage (NLS) for Business Warehouse What is Hot and Cold Data?
Hot Data is permanently accessed with extremely short acceptable response
time.
Dashboard style information continuously updated.
Tactical level data.
Aberrations must be recognized with minimal latency to identify disruption and react.
Warm Data is occasionally accessed with a somewhat higher acceptable
response time
Kept as non-active data in primary BW database. Loaded into memory on demand, after
which full query performance is available
Cold Data continues to be relevant but is less frequently accessed or has a
more relaxed response time constraint.
Paper Report style information evaluated on demand.
Strategic level data.
Long-range evaluation for long-term decisions.
Database Platforms – Best Practices and What’s New Outline Agenda
Near-line Storage (NLS) for Business Warehouse What is Hot and Cold Data?
24 UKI SAP User Group 2013
Near-line Storage (NLS) for Business Warehouse Why would you use a SAP Sybase IQ based NLS with BW on SAP HANA?
SAP HANA platform can deliver the best possible Business Analytics
performance. There are, however, limitations.
Hard Limitations
In- Memory database size is restricted by (combined) RAM size of server machine(s).
Dynamic growth of data size may be faster than your reaction speed in improving hardware.
Soft Limitations
You may not want to cover the cost for the best possible technology for all of your Big Data.
You may want to protect your system from data growth by only keeping a moving time
window In- Memory.
You may want to pursue not the best possible performance but the best value for money.
Split your data between subsets demanding the best possible query
performance and subsets less performance critical.
For non-compressing DBMS, NLS will be more space efficient.
Database Platforms – Best Practices and What’s New Outline Agenda
Near-line Storage (NLS) for Business Warehouse SAP Sybase IQ based NLS with BW on SAP HANA
25 UKI SAP User Group 2013
Near-line Storage (NLS) for Business Warehouse Overview – BW on SAP HANA Example
In-Line Storage
Hot current data
Limited size
Rapid real-time access
Near line Storage
Cold Historical data
Massive volumes
Slightly slower access
SAP HANA
In-memory
Database
High Value,
Current Data
(- OR - OTHER
SUPPORTED
DBMS)
Sybase IQ
Disk Based
Database
Historical Data
Trend Analysis
SAP NetWeaver
Business
Warehouse
Database Platforms – Best Practices and What’s New Outline Agenda
Near-line Storage (NLS) for Business Warehouse Solution Overview
26 UKI SAP User Group 2013
Near-line Storage (NLS) for Business Warehouse How does NLS work?
Beside the primary BW database connection, a second connection to the SAP
Sybase IQ database is maintained
NetWeaver maintains and executes the aging rules
Aging happens along the time axis
Controls data transfer to the NLS instance
Cleans up data in the primary database
Query federation is handled by NetWeaver
Transparent to BW application logic
Evaluation of InfoProviders / DSOs is directed to primary connection, NLS connection or
both
Intermediate results are consolidated by NetWeaver where required
Database Platforms – Best Practices and What’s New Outline Agenda
Near-line Storage (NLS) for Business Warehouse How does NLS work?
27 UKI SAP User Group 2013
Near-line Storage (NLS) for Business Warehouse How does NLS work? – Simple Example
1. Data partitioned
by fiscal
Quarter/year
2. Partitions being moved to
Near-Line Storage are
locked for update
3. Once the data is moved to
Near-Line Storage, it is
deleted from BW
Database Platforms – Best Practices and What’s New Outline Agenda
Near-line Storage (NLS) for Business Warehouse How does NLS work?
28 UKI SAP User Group 2013
Business benefits Measurable success
Reduced SAP NetWeaver BW downtimes
Reduced SAP NetWeaver BW maintenance efforts
Reduced amount of data residing within SAP
NetWeaver BW
Faster querying and reporting
Reduced total cost of ownership for SAP HANA
IMPORTANT NOTE:
Traditional DBMS based BW implementations are
supported by the RDS.
Near-line Storage (NLS) for Business Warehouse Rapid Deployment Solution (RDS) - The Value Proposition
SAP NetWeaver BW Near-line Storage rapid-
deployment solution will:
Help gain a high performing near-line storage with
the functionalities of reporting, ETL, lookup
Reduce your total cost of ownership for SAP
NetWeaver BW (powered by HANA) by
implementing a clear and scheduled data aging
strategy
Enable faster online querying and reporting
Avoid further interfaces to near-line storage as
data is accessed by SAP NetWeaver BW
seamlessly
Decrease maintenance efforts for SAP NetWeaver
BW
Decrease system downtimes for SAP NetWeaver
BW
Near-line Storage (NLS) for Business Warehouse How does NLS work?
Database Platforms – Best Practices and What’s New Outline Agenda
Near-line Storage (NLS) for Business Warehouse Rapid Deployment Solution (RDS) Overview
29 UKI SAP User Group 2013
Near-line Storage (NLS) for Business Warehouse Rapid Deployment Solution – Migration to NLS – Options Overview
Option 1
Option 2
Database Platforms – Best Practices and What’s New Outline Agenda
Near-line Storage (NLS) for Business Warehouse Migration to NLS – RDS Options
30 UKI SAP User Group 2013
Near-line Storage (NLS) for Business Warehouse Rapid Deployment Solution - Go live in as little as 12 weeks!
A special step-by-step guide describes each activity
during the deployment
Service Name: Rapid deployment of SAP NetWeaver BW Near-Line Storage
Implementation Duration: 12 Weeks*
* Based on scope
** Provided as option and impacts overall efforts of the service
Core Offerings
Implementation of SAP NetWeaver BW Near-
Line Storage
Identification of objects to be archived
Installation and Configuration of SAP
Sybase IQ database
Installation of SAP Sybase IQ Client and
configuration of SAP NetWeaver BW to
Near-Line Storage
Create up to 10 DAPs (Data Archiving
Processes)
Create up to 2 Process Chains
Adjust settings for up to 10 Queries and 2
multiproviders
Optional Task**
Support Package Update
Database Platforms – Best Practices and What’s New Outline Agenda
Near-line Storage (NLS) for Business Warehouse RDS - Go live in as little as 12 weeks!
31 UKI SAP User Group 2013
Near-line Storage (NLS) for Business Warehouse
Rapid Deployment Solution – SAP Service Marketplace Database Platforms – Best Practices and What’s New Outline Agenda
Near-line Storage (NLS) for Business Warehouse RDS – SAP Service Marketplace