36
Oracle9i Data Warehouse Review Robert F. Edwards Dulcian, Inc.
Oracle9i Data Warehouse Review
Robert F. EdwardsDulcian, Inc.
Agenda
uOracle9i ServeruOLAP Server
uAnalytical SQLuData MininguETL
uWarehouse Builder 3i
Oracle 9i ServerOverview
u 9i Server = Data Warehouse Platformu DW Core Requirementsu Oracle 9i Server ComponentsØ Relational DatabaseØ OLAP Services – Added in 9iØ Analytical SQL – New functions added in 9iØ ETL Infrastructure – Added in 9iØ Data Mining – Added in 9i
u Core of E-Business Intelligence Platform
CWM compliant MetadataOracle Repository
BusinessIntelligence
Portal
Tools
E-Business IntelligencePlatform
Oracle9iRelational DB
OLAPAnalytical SQL
ETLData Mining
Web Data
ExternalData
ERP/CRMData
OperationalData
Warehouse Builder 3i
Oracle9i Application
Server
Source: Oracle
DW Core Requirements
uPerformance – Most important requirementØQuery Response Time - Critical
uScalability – Growth: Data and UsersØData Growth – very large data volumes
ØAdditional Users – many users supported
uManageability – Simple to maintainØGrowth does not require additional resources
PerformanceEnhancements
u Bitmap Join Index – Spans multiple tables
u Enhanced Materialized Views - Query Rewrite
u Full Outer Joins
uWith Clause – Complex Queries & Sub-queries
u Adaptive Direct I/O Operations – Dynamic I/O
u Automatic Memory Tuning – Runtime Memory
ScalabilityEnhancements
u List PartitioningØ List of discrete valuesØ Precise control over data loaded into partitions
u Parallel QueriesØ Finer grain of parallel query distributionØ Enhanced dynamic load balancingØ Inter-node parallel query improved§ Clustered systems§ MPP platforms
ManageabilityEnhancements
u Appropriate Resource Allocation to QueriesuMaximization of Throughputu DBAs and Users can view status of active jobsu DBA Specified Abort and Re-Queuingu DB Resource Manager (introduced in Oracle8i)Ø Resource Consumer Groups (RCG)Ø Limit Active Sessions per RCGØ Query Governing per RCGØ Criteria-based Change of RCG
Oracle9i Server Overview
u 9i Integrates four data engines:Ø Relational Database – Always here!Ø OLAP Server – previously just Oracle ExpressØ ETL Infrastructure – Custom built or third party toolØ Data Mining – previously just Oracle Darwin
u Analytical SQL Functionsu Java APIs – build complex transforms and queriesu Enhancements Open to Software VendorsØ Implemented in SQL
u Tight integration with Warehouse Builder
OLAP Overview
u On-Line Analytical ProcessingØ Scalable, High Performance Calculation EngineØ End User Business Intelligence Analysis ToolØ High-level aggregates and customized data needs
u OLAP on Oracle9i ServerØ All data resides on the relational databaseØ Most queries satisfied with Analytic SQL functionsØ Java API supplements SQL functionsØ Administration by 9i Enterprise Manager
OLAP Server
uOLAP ServicesØArchitecture and Support
Ø Java OLAP API
Ø Java Development Environment
Ø Express-to-9i Migration Plan
ØAnalytical Functions
ØOLAP SQL Functions
Architecture and Support
u Fully Integrated with 9i database
u Relational Database is OLAP Data StoreØ Relational - not a proprietary array structure
Ø Scalable - very large volumes (Terabytes) supported
Ø Lower Overhead than Multi-Dimensional DB
u Administration with Oracle Enterprise Manager
u Data Modeling with 9i Dictionary and Meta Data
u High Availability and Security support by 9i
Java OLAP API
u Java OLAP API provides Object Orientedenvironment for building complex analyticalqueriesØ Multi-dimensional object modelØ Full range of functions to support analytical AppsØ Supports asymmetrical queries (nested rankings)Ø Support for calculated dimension members
and measures (facts)Ø Incremental query constructionØ Multi-dimensional cursors
u Supplements analytical SQL functions
Development Environment
u Business Intelligence (BI) BeansØ Analytically aware building blocks -
§ Tables, cross tabs, graphs, query & calc builders
u JDeveloper 9i environmentu Runtime Repository for Analytical ObjectsØ Supports collaborative analysis
u Data Dictionary and Oracle Repository meta dataØ Fully describe multidimensional analytical model
uMeta Data administered by OEM and OWB
OLAP Migration Plan
uExpress and DB converge: OLAP ServicesuSupport continues for Express 6.xØMaintenance & compatibility with 9i DB
Ø Financial and Sales Analyzer continue support
uNew features added to 9i OLAP Services
uNew Analysis and Planning app for 9iuMigration tools will be available
Analytical SQL
u Introduced in Oracle8/8iuAdditional functions in Oracle9i
uSupport for OLAPØAggregations, including Grouping functions
uSupport for Data MiningØ Sampling and Statistics
uMost functions added to ANSI SQL
Analytical Functions – 8i
u Introduced with Oracle8i:Ø RankingØMoving Window AggregatesØ Period over Period ComparisonsØ Ratio to ReportØ Statistical Functions –§ Linear Regression§ Correlations
Analytical Functions – 9i
u Introduced with Oracle9i:Ø Inverse Percentiles
ØHypothetical Rank and Distributions
ØHistograms
Ø First / Last Aggregates
ØGrouping Sets
Ø Composite Group By (column sets)
OLAP SQL Functions
u Aggregation FunctionsØ 8.0 – Cube and Rollup (Group By)Ø 9i – Grouping Sets§ Grouping Sets – Sets of aggregation§ Concatenated Grouping Sets –
Set Combinations (like Cartesian product)§ Composite Columns in Group By –
Column sets are treated like a single unit
Data Mining
uData MiningØDiscover hidden patterns and
relationshipsØMachine-learning Mining AlgorithmsØScoring ModelsØJava API InterfaceØAvailable in 9i EE
Data Mining
uOracle Data MininguMachine-learning Algorithms
uModel-based ScoringuJava Functions
Oracle Data Mining
u Oracle buys Darwin and Thinking Machinesu Initially Unix-based and stand alone, flat file I/Pu Data Mining Suite (Darwin 3.7)Ø Accesses database, network able, wizards introduced
u Integrated with CRM 11i – Marketing applicationu Architecture Changes -Ø Shift from stand alone to database platformØ Access to database and networkØ Unix Interface to Windows GUIØ Access via Web browser
Data Mining Algorithms
uMachine-learning Data Analysis techniquesu Classification and PredictionØ Naïve Bayes algorithm – supervised learningØ Binary and Multiple-class outcomes
u Association RulesØ Detect co-occurring events hidden in dataØ Unsupervised learning
u Results of analysis are built as models used toscore new data
Model Scoring
uScoring – Process of Predicting OutcomesuNew data scored using Naïve Bayes modelsØ Batch – applied to a table of records
§ Prediction stored in another table
ØOn-demand – applied to single record
§ Prediction returned to calling application
Java Functions
uJava API is used for development of all datamining functionsØData preparation
ØModel Building
ØModel-based Scoring
uJava Data Mining (JDM)Ø Emerging industry data mining standard
ETL Overview
u Extraction, Transformation & Loading
Ø Extracts source data from database tables and flat files ofoperational applications
Ø Transforms (converts) source data to a consistent formatusable by the data warehouse and BI apps
Ø Loads transformed data to the warehouse tables
Ø Builds aggregates and customized data requirements fordata marts and OLAP servers
u Formerly had to be built in-house using PL/SQL or otherlanguages, or use a third party tool
u Data Migration consumed extensive project resources
u Warehouse Builder is Oracle’s ETL development tool
Oracle 9i ETL
u Scalable ETL infrastructureuMajor new features / enhancementsØ Change Data CaptureØ External File AccessØUpserts and Multi-table InsertsØ Table FunctionsØ Transportable Tablespaces
u Leveraged by Warehouse Builder 3i design anddeployment functions
ETL Enhancements
uChange Data Capture – detects changes insource dataØ Replication and LogMiner
uExternal Tables – access flat files with SQLlike DB tables, without staging
uUpserts – single step update and insertuMulti-table Inserts – single step inserts to
multiple tables
ETL Enhancements (2)
u Table Functions – complex transformationsØ Complex functionality not available with SQL aloneØ Implemented in PL/SQL, Java, C, C++Ø Pipelined, fully parallelized, and scalable
u Transportable Tablespaces – accommodatesmultiple block sizes, eases data movement fromOLTP to warehouseØ Block size can be defined for each tablespaceØ OLTP – 4-8 KB blocksØ Data Warehouse – 16-32 KB blocks
Warehouse BuilderOverview
uWarehouse Builder is Oracle’s DW designand ETL development tool
uDesign, deploy and manage warehouses,data marts, and business intelligence apps
uTightly integrated with Oracle 9i ServeruMaintains Oracle Repository (Meta Data)
uLeverages Oracle9i’s ETL infrastructure
Warehouse Builder 3i
uOWB 3i is a major releaseuNew Mapping and Transformation
paradigm
uNew Mapping ArchitectureuMajor Enhancements
3i Mapping and TransformFeatures
u Join Operator – join multiple sourcesu Split Operator – output to multiple targetsuMulti-stage Transformation – pipeline SQL expr.uMultiple Targets – multiple tables in single passu Key Lookup – lookup and replace valuesu Set Operator – support for Union, Minus, Intersectu Inline Expressions – SQL expression in Selectu Incremental Code Generation – generate and view
code up to any point within the data flow
3i Mapping Architecture
u Previous architecture (OWB 2.x)Ø Two level maps – High-level and DetailedØ Source and target dependency for transform maps
u New architectureØ Connectivity through parametersØ Transforms are independent of sources and targets
u Operator: output is a subset of input rowsØ AGGREGATES and FILTERS
u Transform: output is equal to the input rowsØ Standard SQL functions, e.g. TO_CHAR
3i Enhanced Features
u Enhanced flat file IntegratoruMeta Data Reconciliation –source and target DBsu Upgrade or drop warehouse schemau Complete inline SQL expression builderuMultiple-user support on OWB repositoryu International Name and Address Cleansingu Reverse Engineering PL/SQL into repositoryu Total Data Warehousing Managementu Significant performance enhancements
Summary
u 9i Server: A True Data Warehouse PlatformØ Integrated DB, OLAP, ETL and Data Mining engines
u Core of E-Business Intelligence PlatformØ Warehouse Builder / Oracle9i / 9i Application Server
u All data stored and processed on the relational DBØ ETL support for DW and Data MiningØ Analytical SQL and Java API support OLAP and DMØ Internet/Web enabled, with JDeveloper environment
u Warehouse Builder 3iØ New Mapping and Transform paradigmØ Major Enhancements
