OBIEE11g Seminar by Mark Rittman for OU Expert Summit, Dubai 2015

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

OBIEE11g Update and New Features Mark Rittman, CTO, Rittman Mead Oracle University Expert SummitDubai, February 2015





About the Speaker

•Mark Rittman, Co-Founder of Rittman Mead •Oracle ACE Director, specialising in Oracle BI&DW •14 Years Experience with Oracle Technology •Regular columnist for Oracle Magazine •Author of two Oracle Press Oracle BI books •Oracle Business Intelligence Developers Guide •Oracle Exalytics Revealed •Writer for Rittman Mead Blog :http://www.rittmanmead.com/blog

•Email : [email protected] •Twitter : @markrittman

http://www.rittmanmead.com/blog

mailto:[email protected]







About Rittman Mead

•Oracle BI and DW Gold partner •Winner of five UKOUG Partner of the Year awards in 2013 - including BI •World leading specialist partner for technical excellence, solutions delivery and innovation in Oracle BI

•Approximately 80 consultants worldwide •All expert in Oracle BI and DW •Offices in US (Atlanta), Europe, Australia and India •Skills in broad range of supporting Oracle tools: ‣OBIEE, OBIA ‣ODIEE ‣Essbase, Oracle OLAP ‣GoldenGate ‣Endeca



OBIEE 11g: Where Are We Now?

•OBIEE 11.1.1.7 release came out in April 2013 - many new features + updated look and feel •Fourth major release of OBIEE 11g since the initial launch back in 2010 •Enterprise BI platform centered around the Common Enterprise Semantic Model (RPD) •Mobile BI apps, MS Office integration, ad-hoc, dashboard and published reporting

•Built around Oracle Fusion Middleware •Deployable on Windows, Unix, Linux •Accessing a range of enterprise data sources ‣Oracle and other RDBMSs ‣Essbase and other OLAP servers ‣Files, XML, web services ‣ADF and SOA sources ‣TimesTen in-memory database



Embedded and Leveraged Throughout the Oracle Tech and Apps Stack

•Packaged EPM and CRM analytics through Oracle Business Intelligence Applications 11g •Embedded in the Oracle Fusion Applications •Available as BI components within Oracle Fusion Middleware



Part of the Wider Oracle BI Foundation Suite 11g

•Typically now sold packaged as Oracle BI Foundation Suite 11g ‣Includes Oracle BI EE (dashboard analytics + enterprise semantic model) ‣Oracle BI Mobile ‣Oracle Scorecard & Strategy Management ‣Oracle Essbase Server

•This provides the context, and the driver, forintegration between Essbase and OBIEE ‣Essbase is the “go-to” OLAP server for Oracle BI EE, and Oracle’s analytic apps ‣It’s also the standard software bundlewhen licensing Oracle Exalytics



Oracle’s Strategy for Business Analytics

•Connect to all of your data, from all your sources, using integrated tools •Deliver analytics on-premise, or in the cloud •Package solutions for known problems and fixed sources, and •Engineered systems to lower TCO and improve query performance and user experience

In The Cloud

On-‐Premise

Any Data

Packaged Analytic and EPM Apps

Engineered Systems

Integrated Tools



Recent Product Releases



And Coming Soon with OBIEE 12c …



New in OBIEE 11.1.1.7 - Updated Look & Feel, Visuals, Charts

•New FusionFX look-and-feel •View Suggestion Engine •Performance Tiles •Breadcrumb navigation links •All new standard Views, Print layouts, Dashboard layouts

•Map Lines, Map Targets, Non BI Layers…

•Blob images content integration •Mobile optimized pages



New in OBIEE 11.1.1.7 : Full Integration of Essbase into OBIEE Tech Stack

•From OBIEE 11.1.1.7, Essbase is now an install option within the OBIEE product installer ‣Has to be licensed separately, or as part of BI Foundation

• Installs Essbase Server, Essbase Studio, Financial Reporting and other BI-related/complementary tools alongside OBIEE

•Management of Essbase Server, Security, start/stop etcall from single Enterprise Manager farm



Essbase and OBIEE within Oracle BI Domain - EM Management

•Essbase can be stopped, started, restarted from EM via OPMN •Essbase metrics reported on from EM • Integrated security tools via OPSS application roles and policies



The Return of EPM Workspace Integration from OBIEE 11.1.1.7

•EPM Workspace integration returns with OBIEE 11.1.1.7! •SSO via shared OPSS (FMW) security •Launch BI content from within Workspace •Store FR reports in Catalog (must launch fromEPM Workspace though, not from OBIEE dashboard)



Also - Full Essbase Cube Spin-Off in OBIEE 11.1.1.7

•More of a developer preview at this stage, but ... • ... OBIEE logical SQL command CREATE CUBE providesability to create whole Essbase ASO or BSO cube from RPD

•Rather than just a single aggregate slice •One for the future - full OLAP data mart automation?



Essbase Role within Oracle BI Foundation Suite

•Essbase is deployed alongside OBIEE to complement OBIEE’s existing relational capabilities •Adds an enterprise-class OLAP server to the stack, deeply integrated •Essbase is supporting BI; elsewhere, in EPM Suite, it runs the show alone •All integration work in this context is about making BI run better,not replacing Essbase-specific tools or removing EPM Suite



And for a Bonus: SmartView Compatibility across Essbase and OBIEE

•From OBIEE 11.1.1.7, SmartView can now be used with OBIEE for Office integration •Ability to create new reports as well as analyze Catalog content via Excel, Word etc •Replaces BI Office (and original SmartView from OBIEE 10g)



Introducing Oracle BI Applications 11g and ODI 11g

•All-new, 11g release of the BI Applications •New content, applications, uptakes OBIEE 11g visuals etc •Simplified topology - now uses ODI 11g for the ETL ‣Informatica support coming soon

•No DAC - uses web-based configuration tools, and ODI, to control and orchestrate the ETL

•Lower TCO, faster ETL, simpler architecture



New Content in BI Apps 11.1.1.7.1

•Two new analytic applications: ‣Student Information Analytics ‣Indirect Spend Analytics

•New content for existing applications ‣Financial Analytics ‣HR Analytics ‣Project Analytics etc

•Dashboards and analyses updated for 11g ‣Performance tiles ‣Trellis charts ‣Hierarchical columns etc



New in BI Apps 11.1.1.8.1 - Additional Content



Use of Oracle Data Integrator 11g as Embedded ETL Engine

•Oracle Data Integrator is now an option for the BI Apps embedded ETL engine

•Potential TCO benefits + use of Oracle strategic tool •DAC also now gone, replaced by web-based config •Same concept of source adapters, BI Apps DW •New option to use GoldenGate to replicate sourcedata into SDS (source dependent schemas)

•Future 11.1.1.7.x releases will also support Informaticafor ETL, Fusion Apps as a data source



Use of ODI for Embedded ETL

•Load configuration tasks now carried out usingBI Configuration Manager + Functional Setup Manager ‣Web-based admin tools with closer integration to ODI

• Informatica Execution Plans now replaced by ODI Load Plans ‣Load Plan Generator application assembles plans based onsubject area selection, and task dependencies

•ODI Studio + Console now used for ETL customization + mgmt •Potential benefits from closer tool integration, lower TCO,more efficient ETL processing (ELT)



BI Applications Product Roadmap



Runs Better on Oracle Exalytics In-Memory Machine

•Engineered System for running Oracle’s BI and EPM products •1TB RAM, 40 CPU cores, Oracle Linux 64-bit OS •Optimizations to move moreof BI + OLAP processing in-memory

•Lower TCO through co-ordinated patching, support,software releases

•Comes with OBIEE, Essbase (ASO + BSO),TimesTen in-memory cache

Exalytics In-Memory Machine

Spans Relational, Multi-Dimensional, and Unstructured analysis, combined with Financial & Operational Planning ‣ In-Memory Optimized Hardware ‣ In-Memory Oracle BI, TimesTen, Essbase, and Endeca ‣ Many In-Memory Software Innovations

Tightly-Integrated with Exadata, and Big Data Appliance



Exalytics Also Now Supports Essbase, and Endeca Information Discovery

• In-Memory Essbase for planning, budgeting and sales analysis-style OLAP applications •Endeca Information Discovery for search/analytic applications against diverse data

In-Memory Cache

Essbase Planning Engine

Smart Storage Manager

Lock Manager

Unified Indexing

Data Mashup

Text Analysis

Unified Search

Faceted Navigation

Interactive Exploration

Information Discovery

Oracle Exalytics

In-Memory Machine



Oracle Database 12cR1

•Latest release of Oracle Database, focused on the cloud •Headline feature - Container and Pluggable Databases •New DB Express Enterprise Manager utility + many new general DB new features



Oracle Database 12c In-Memory Option

•Became available in 2014, option for Oracle Database 12c Enterprise Edition •Brings column-store and in-memory capabilitiesto the core Oracle Database

•Now an option for use with Oracle Exalytics •Rittman Mead were beta testers for 12c In-Memory,quoted at Larry Ellison product launch….



Database In-Memory Option - Exalytics Certification

•Exalytics-specific installation/configuration ‣Pre-configured to place all data in-memory – no need for special “alter table inmemory”

•Works with OBIEE 11.1.1.7 ‣Caching data marts – all queries will work! ‣Certified with all ETL tools as Oracle DB ‣Summary advisor works with DBIM, just like ‣with TimesTen, to create optimal aggregates

•Supported on all existing Exalytics hardware models •Sizing: 50%-60% of Exalytics memory can be • reserved for data storage



Now Two Options for In-Memory Exalytics Cache



Deploying OBIEE in the Cloud

•Many OBIEE customers are considering deploying their BIinfrastructure in the “cloud”

• “Push” and “Pull” factors in-play •Perceived benefits in reducing amount of on-premise servers,software installs, upfront-license costs

•May also be prompted by migration of rest of IT infrastructureto the cloud

•To enable business agility - deploy the system now -And potentially reduce dependency on IT department



Oracle Cloud Business Intelligence

•BICS v1 available since September 2014 •Part of the wider PaaS (platform-as-a-service)offering including database, Java, documents etc

•Aimed at a very specific set of use-cases initially •Adding BI and analytics to cloud-based apps •Departmental sandbox-type analysis



Self-Service, Consumer-Style Experience



Oracle BI Cloud Service v1 Functionality



Oracle Cloud Business Intelligence - What is it

•Runs in Oracle’s public cloud environment (http://cloud.oracle.com) •No upfront license cost

•Pay monthly as OpEx •Only reports against a single Oracle database schema (Oracle Cloud Schema-as-a-Service)

• “Standard” OBIEE - but will possibly preview newfunctionality before on-premise versions

•Patched automatically by Oracle •Customer able to be max one version behind?

•Comes with two instances - pre-prod and prod •Code promotion through BI Archive export/import

Cloud

On-PremiseAdmin accessover HTTP

Oracle Schema-as-a-serviceDatabase (single source)Oracle BI-as-a-Service

http://cloud.oracle.com



Oracle Cloud Business Intelligence - What’s new?

•New component: Thin Client Data Modeller •Replacement for OBIEE On Premise Admin Tool

•Simplifies Metadata Layer creation and management

•Accessible via browser •No need of Windows only client •Contains a subset of the features

•Preview of New Visualisations



The Oracle Reference DW Architecture c. 2013

•Centred on the Oracle RDBMS and structured data •All incoming data is modelled at load point, schemas assigned, stored in RDBMS layers •BI metadata layer and ability to federate at BI query stage •Data storage capacity limited by RDBMS





The Oracle Data Warehousing Platform - 2014



Combining Oracle RDBMS with Hadoop + NoSQL

•High-value, high-density data goes into Oracle RDBMS •Better support for fast queries, summaries, referential integrity etc

•Lower-value, lower-density data goes into Hadoop + NoSQL ‣Also provides flexible schema, more agile development

•Successful next-generation BI+DW projects combine both - neither on their own is sufficient



Productising the Next-Generation IM Architecture



OBIEE for Enterprise Analysis Across all Data Sources

•Dashboards, analyses, OLAP analytics, scorecards, published reporting, mobile

•Presented as an integrated business semantic model •Optional mid-tier query acceleration using Oracle Exalytics In-Memory Machine

•Access data from RBDMS, applications, Hadoop, OLAP, ADF BCs etc

Enterprise SemanticBusiness Model

Business PresentationLayer (Reports, Dashboards)

In-Memory Caching Layer

ApplicationSources

Hadoop /NoSQL Sources

DW / OLAP Sources



New in OBIEE 11.1.1.7 : Hadoop Connectivity through Hive

•MapReduce jobs are typically written in Java, but Hive can make this simpler •Hive is a query environment over Hadoop/MapReduce to support SQL-like queries •Hive server accepts HiveQL queries via HiveODBC or HiveJDBC, automaticallycreates MapReduce jobs against data previously loaded into the Hive HDFS tables

•Approach used by ODI and OBIEE to gain access to Hadoop data •Allows Hadoop data to be accessed just like any other data source (sort of...)



Coming Soon - Oracle Big Data Discovery



Seminar Agenda

•Day 1 ‣OBIEE11g Update and New Features ‣Adding Exalytics and In-Memory Analytics to OBIEE 11g ‣OBIEE Development & Deployment Best Practices ‣No Silver Bullets - OBIEE 11g Performance in the Real World

•Day 2 ‣Oracle BI Cloud Service Overview, Tips and Techniques ‣Moving to Oracle BI Applications 11g + ODI ‣Oracle Essbase and Oracle BI EE 11g Integration Tips and Techniques ‣OBIEE 11g and Predictive Analytics, Hadoop & Big Data



OBIEE11g Update and New Features Mark Rittman, CTO, Rittman Mead Oracle University Expert SummitDubai, February 2015



Adding Exalytics and In-Memory Analytics to OBIEE 11gMark Rittman, CTO, Rittman Mead February 2015



The Benefit of In-Memory Analytics for OBIEE & Essbase

• In-Memory analytics - lightening-fast response, free-form analysis and aggregation •Rich, immersive interactive dashboards powered-by high-spec hardware

•Extra OBIEE + other features only available on this platform

•Enables fast development controlled by the business

•Faster planning and budgeting



Oracle Exalytics In-Memory Machine

•Engineered system, complements Oracle Exadata Database Machine (though can work standalone)

•Combination of high-end hardware and optimized versions of Oracle’s BI, In-Memory Database and OLAP software

•Delivers “in-memory analytics” focusing on analysis, aggregation and UI ‣Rich, interactive dashboards with split-second response times ‣1-4TB of RAM, to run your analysis in-memory ‣Infiniband connection to Exadata and Oracle Big Data Appliance ‣40-60 CPU cores to support high-levels of user concurrency ‣Lower TCO through known configuration, combined patch sets ‣Contains software features only licensable throughExalytics package



Exalytics as the Query Performance Enhancer

• In conjunction with a well-tuned source data warehouse, Exalytics adds an in-memory analysis later

•Based around Oracle TimesTen for Exalytics, Oracle’s In-Memory Database, and now Oracle Database 12c In-Memory Option

•Aggregates are recommended based on query patterns, and automatically created in TimesTen or Oracle Database In-Memory Options

•Summary Advisor makes recommendations, THAT adapt as queries change

•Meant to be “plug-and-play” - no need for expensive data warehouse tuning

•So how does it work in-practice?

TimesTen BI Server

Exal

ytic

s

Aggregates

Data WarehouseDetail-level

Data



Summary Advisor for Aggregate Recommendation & Creation

•Utility within Oracle BI Administrator tool that recommends aggregates •Bases recommendations on usage tracking and summary statistics data •Captured based on past activity •Runs an iterative algorithm that searches,each iteration, for the best aggregate



Recommendations Based on Enhanced Usage Tracking Data

•Historically, usage tracking has been tracked using S_NQ_ACCT ‣Holds basic usage tracking statistics + logical SQL query

•Now supplemented by S_NQ_DB_ACCT ‣Extra usage tracking information, includes physical SQL

•Exalytics Summary Advisor uses S_NQ_SUMMARY_ADVISOR ‣Contains summary statistics, execution time etc ‣Gathered at same time as usage tracking whenExalytics is enabed ‣Contents can be derived from usage tracking if needed



Oracle TimesTen for Exalytics

•New version of TimesTen specifically for Exalytics (and only available with Exalytics) •Support for analytic functions ‣Perform all the processing at source ‣Combine with being in-memory = should be very fast

•Column compression ‣Whitepaper cites 5x - possible for dimension tables, usually much lower for fact tables (20% compression) ‣Given the hardware capacity, we could seriously contemplate loading the whole Data Warehouse into memory - see techniques and limitations later on ‣Opens up lots of interesting design potential

•We can load aggregates into TimesTen, leave base data at source, and use OBIEE’s Vertical Federation capability to seamlessly report across both ‣All hidden from the end-user, all they will know is that their reports run fast!



TimesTen and OBIEE Architecture

•Oracle BI Server communicates with TimesTen through TimesTen Client •Summary Advisor, and nqcmd use Oracle BI Server to access TimesTen •Typical single TimesTen database per Exalyics machine ‣Max TimesTen database size around 300MB

-Due to need to set aside equalTemp size for the Perm size selected

•Clustered Exalytics boxes can be daisy-chainedtogether using InfinBand connections ‣For HA scenarios, does not increase available RAM ‣Summary advisor scripts write to both TimesTendatabases, replicating aggregates ‣TimesTen databases can be “wired together”for failover/HA purposes

TimesTen Memory-ResidentDatabase

CheckpointFiles

Log Files

ODBC

Oracle BIServer

nqcmdSummaryAdvisor



Example : Using Exalytics to Pre-Aggregate and Cache BI Apps Data

•Can we use the Summary Advisor to automatically cache commonly-used aggregates in-memory? ‣Similar to regular OBIEE caching, relies on query repeatability + use of aggregation

•Could we copy all, or part, of the BI Apps data warehouse directly into TimesTen? •How would we update the RPD to point to the in-memory tables? •How fast would TimesTen be to load, and to query, vs. Oracle/SQL Server/DB2 etc? •Here’s our thoughts and R&D to date....



Using the Summary Advisor on BI Apps Step 1 : Start Advisor

•Server has to be an Exalytics server, in this example is patched-up to 11.1.1.6.9 •Workstation has the 11.1.1.6.9 BI Administration tool installed •Select Tools > Utilities, then Oracle BI Summary Advisor from utility list

12



Using the Summary Advisor on BI Apps Step 2 : Scope Source Queries

•By default, all queries registered in the usage tracking and summary statistics tables are in-scope

•Refine the recommendations by limiting timeframe, and setting minimum accumulated time threshold

•Still an opportunity later on to pick and choose from recommended aggregates •Once selected, then select the TimesTen connection pool and database as the aggregate table target

34



Using the Summary Advisor on BI Apps Step 3 : Create Aggregates

•Summary Advisor then recommends a set of “candidate” aggregates, which you can choose to implement

•Select all, none or some of the recommended aggregates •Then run the resulting logical SQL script using the nqcmd utility •Note - may need to clean-up BI Apps DW data to remove duplicates etc before script completes OK



Using the Summary Advisor on BI Apps Step 4 : Review RPD and Data

•Aggregate Persistence process called by the “create aggregates” process also maps tables in RPD

•Physical layer contains entries for the TimesTen tables •Business Model and Mapping later contains vertically-federated LTSs for the new TT tables



Summary Advisor for BI Apps : Pros and Cons

•Pros ‣Automatically analyzes query patterns and recommends aggregates to accelerate them ‣Handles the registering of the TimesTen tables in the RPD, including mapping back into business model ‣Supports any data source that the BI Server supports

•Cons ‣Queries have to have run before they’ll be considered for loadinginto TimesTen for Exalytics ‣Relies on subsequent queries being able to use those aggregates ‣Could get unwieldy if many aggregates are registered in the RPD ‣Summary Advisor process does not automatically clear downtables that don’t feature in future recommendations ‣Refresh process for aggregates is inefficient



Refreshing Summary Advisor Aggregates Within the TimesTen Data Mart

•Aggregates are built in TimesTen to support sub-second response times •Summary Advisor tool suggests aggregate dimensionality and grain, generates script •BI Server’s Aggregate Persistence executes script:

1. Create TimesTen aggregate table 2. Populate TimesTen aggregate 3. Update RPD online with new aggregate metadata

•Handles supporting dimensions tooBase data Aggregates

TimesTen

OBIEE Aggregate Persistence

RPD

Aggregate tables created

RPD updated with new aggregate mappings

Aggregate tables loaded from base data



Out-of-the-Box Summary Advisor Limitations

•To refresh an aggregate, Summary Advisor deletes and rebuilds from scratch •The RPD is edited directly on the BI Server each time an aggregate is created or rebuilt •Build failures can be difficult to debug, if it fails can leave the RPD in an inconsistent state with TimesTen

After a failed build, the aggregates are still in the RPD, but no longer exist in TimesTen

nqquery.log suggests possible errors but no clear root cause

This is that the error the user sees



Alternatives to using Aggregate Persistence

•However aggregates are refreshed, they must be included in the RPD ‣Can be done manually ‣Aggregate Persistence is useful for this

•Complete refresh of aggregate data ‣Extract the SQL that OBIEE generates in Aggregate Persistence, run this through ODI ‣Write bespoke aggregate refresh code in ODI

• Incremental refresh using GoldenGate and ODI ‣Instead of rebuilding aggregates in their entirety each time, only update the part of the aggregate that has changed on the base data



Custom BI Server ETL Option : Incremental BI Server Refresh

•Uses BI Server to do the refresh, but invokes just part of it - the data refresh part •Avoids unnecessary aggregate table drop/rebuild, and online RPD edit •Uses BI Server logical ETL SQL features to just refresh the TT aggregate table, using only the latest set of detail-level data ‣Avoids use of additional ETL tools ‣No additional license cost ‣Works against any BI Server source



Custom BI Server ETL Option : Incremental BI Server Refresh

•Uses a number of “undocumented” but support BI Server ETL features ‣POPULATE command for loading data into an RPD table ‣WHERE clause for limiting the refresh to just the incremental load ‣INACTIVE_SCHEMAS to stop BI Server refreshing using existing RPD agg table

•See http://www.rittmanmead.com/2013/04/incremental-refresh-of-exalytics-aggregates-using-native-bi-server-capabilities/ for full details including scripts etc

SET VARIABLE DISABLE_CACHE_HIT=1, DISABLE_CACHE_SEED=1, DISABLE_SUMMARY_STATS_LOGGING=1, INACTIVE_SCHEMAS='"TimesTen Aggregates".."EXALYTICS"'; populate "ag_sales_month" mode ( append table connection pool "TimesTen aggregates"."TT_CP") as select_business_model "Sales"."Fact Sales"."Sale Amount" as "Sale_Amoun000000AD","Sales"."Dim Times"."Month YYYYMM" as "Month_YYYY000000D0" from "Sales" where "Dim Times"."Month YYYYMM" = VALUEOF("THIS_MONTH");

http://www.rittmanmead.com/2013/04/incremental-refresh-of-exalytics-aggregates-using-native-bi-server-capabilities/



Oracle Database 12c In-Memory Option

•Became available in 2014, option for Oracle Database 12c Enterprise Edition •Brings column-store and in-memory capabilitiesto the core Oracle Database

•Now an option for use with Oracle Exalytics •Rittman Mead were beta testers for 12c In-Memory,quoted at Larry Ellison product launch….



Oracle Database 12c In-Memory Option Goals

• 100x Faster Queries: Real-Time Analytics • Instantaneous Queries on OLTP Database or Data Warehouse

• Faster Mixed Workload OLTP

• Transparent: no application changes • Simple to Implement



Dual-Format Database Storage : Row and Column Store

•BOTH row and column formats for same table •Simultaneously active and transactionally consistent

•Analytics & reporting use new in-memory Column format

•OLTP uses proven row format

Memory Memory

SALES SALES

Row Format

Column Format



Leveraging SIMD Vector Instruction for Fast Scanning

•Each CPU core scans local in-memory columns •Scans use super fast SIMD vector instructions •Originally designed for graphics & science •Billions of rows/sec scan rate per CPU core •Row format is millions/sec

Vector RegisterLoad

multiple region values

Vector Compare all values in 1 cycle

CPU

Memory

REGION

CA

CACA

CA

Example: Find all sales in region of CA

> 100x Faster



In-Memory Column-Store Join Optimisation

•Converts joins of data in multiple tables into fast column scans •Joins tables 10x faster than regular row-store on disk

Example: Find total sales in outlet stores

SalesStores

Store ID

Amou

nt

Type=Outlet

StoreID in 15, 38, 64

Store ID

Type

Sum



Objective : Generate Reports Instantly

•Dynamically creates in-memory report outline •Then report outline filled-in during fast fact scan •Reports run much faster without predefined cubes

Example: Report sales of footwear in outlet stores

Sales

Stores

ProductsIn-‐Memory

Report Outline

Footwear

Outlets

$$$

$$$$

Footwear

SalesOutlets



What is the Database In-Memory Option?

•Option on top of Oracle Database Enterprise Edition •Entire data resides on disk •Tables can be selectively brought into in-memory columnar store

•Columnar compression supported •Full Oracle SQL supported •Supported on all hardware



Database In-Memory Option - Exalytics Certification

•Exalytics-specific installation/configuration ‣Pre-configured to place all data in-memory – no need for special “alter table inmemory”

•Works with OBIEE 11.1.1.7 ‣Caching data marts – all queries will work! ‣Certified with all ETL tools as Oracle DB ‣Summary advisor works with DBIM, just like ‣with TimesTen, to create optimal aggregates

•Supported on all existing Exalytics hardware models •Sizing: 50%-60% of Exalytics memory can be • reserved for data storage



Database In-Memory Option - Exalytics (Pricing) Restrictions

•Data must be 100% in-memory ‣Oracle BI uses DBIM as an in-memory cache, just like TimesTen ‣For full-use of DBIM outside of Summary Advisor - requires full-use 12c license

•Single instance only •Not certified for ‣OLTP applications ‣RAC/grid installation

•ASM – only file-based store •Bare Metal Intallation available now ‣Installation on OVM: guidance under development



Now Two Options for In-Memory Exalytics Cache



Choosing Between TimesTen and DB In-Memory Option

•New customers: Choose Database In-Memory ‣Database In-Memory Option is the strategic choice ‣You can leverage existing skillset ‣Supports direct data mart caching; advanced analytics

•Consider TimesTen for Exalytics when ‣You are expanding an existing TimesTen footprint ‣You want a low cost option



Oracle In-Memory Columnar Technology

•Pure in-memory column format •Not persistent, and no logging •Quick to change data: fast OLTP •2x to 20x compression •Enabled at table or partition level •Available on all hardware platforms

SALES

Pure In-‐Memory Columnar



DemoOracle Exalytics, Summary Advisor &Oracle Database 12c In-Memory Option



Manually Configuring 12c In-Memory Option for OBIEE

•As well as using Oracle Database 12c In-Memory Option with Exalytics, you can alsouse standalone with OBIEE, and configure individual tables for in-memory column store ‣Requires full-use license for DB + In-Memory Option, not special Exalytics-only packaging

•Useful for selectively placing tables in-memory, or entire data warehouse

•Example - run SampleApp Airline Delays Appentirely in-memory



SampleApp Airline Delays Dataset

•23 Database tables in 11gR2, some containing GBs of data, facts at transaction level •Complex SQL queries generated from OBIEE analyses •Response-time between 10-50s / analysis

select sum(T255906.Dep_Delay_00039B26) as c1, sum(T255906.Arr_Delay_00039B22) as c2, sum(T255906.Z_of_Fligh00039B28) as c3, substr(T255216.Carrier00039BA9 , 1, 25) as c4, T255216.Carrier00039BA9 as c5, T255216.Carrier_Co00039BAA as c6 from BI_AIRLINES_AGG.SA_16_Dest00039D06 T255357, BI_AIRLINES_AGG.SA_Time_Mo00039CFB T255737, BI_AIRLINES_AGG.SA_31_Carr00039CEB T255216, BI_AIRLINES_AGG.FACT_AGG_OR_06M T255906 where ( T255216.Carrier_Co00039BAA = T255906.Carrier_Co00039BAA and T255357.Dest_Airpo00039C2A = T255906.Dest_Airpo00039C2A and T255737.Dep_Month00039C07 = T255906.Dep_Month00039C07 and substr(T255216.Carrier00039BA9 , 1, 25) = 'SunFlower Airlines' and (T255357.Dest_Regio00039C31 in ('Midwest Region', 'Northeast Region', 'South Region', 'West Region')) and T255737.Month_of_Y00039C0F between 3 and 12 ) group by T255216.Carrier00039BA9, T255216.Carrier_Co00039BAA order by c5, c6, c4

0"

10"

20"

30"

40"

50"

60"

Airline"Delay"Components"

Flight"Delay"Counter"

Delay"Increase"

A>er"Depart"

Delay"increase"A>er"Dept"Unprompt"

Delay"Performance"

Matrix"

Time"Blocks"Delay"

Delay"Cause"Analysis"

Delay"Late"Flight"Cause"

Pie"

Geographical"Analysis"1"

11gR2%DB,%No%Aggs%

11gR2"DB,"No"Aggs"



Sample App Dataset with Aggregate Tables

•Existing Oracle11gR2-based schema can also use aggregates in the RPD •When enabled, brings down 11gR2query response time accordingly

•So can Oracle Database 12c In-Memory Option improve on this? •And are aggs still needed?

0"

10"

20"

30"

40"

50"

60"



Delay"Increase"A>er"

Depart"

Delay"increase"A>er"

Dept"Unprompt"

Delay"Performance"

Matrix"

Time"Blocks"Delay"



Pie"


11gR2"DB,"No"Aggs"

11gR2"DB"with"Agg"Tables"



Enabling Airline Delays Dataset for In-Memory Option

•Export from 11gR2 database, import into 12c Database with In-Memory Option •Enable In-Memory area in SGA, bounce database

SQL> alter system set inmemory_size = 1G scope=spfile; System altered. SQL> show parameter sga_target NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ sga_target big integer 6032M SQL> alter system set sga_target = 7032M scope=spfile; System altered. SQL> shutdown immediate; Database closed. Database dismounted. ORACLE instance shut down. SQL> startup ORACLE instance started. Total System Global Area 7381975040 bytes … In-Memory Area 1073741824 bytes Database mounted. Database opened.



Mark BI_AIRLINES Schema Tables for In-Memory

alter table AIRCRAFT_GROUP inmemory priority high; alter table AIRCRAFT_TYPE inmemory priority high; alter table AIRLINES_USER_DATA inmemory priority high; alter table AIRLINE_ID inmemory priority high; alter table CANCELLATION inmemory priority high; alter table CARRIER_GROUP_NEW inmemory priority high; alter table CARRIER_REGION inmemory priority high; alter table DEPARBLK inmemory priority high; alter table DISTANCE_GROUP_250 inmemory priority high; alter table DOMESTIC_SEGMENT inmemory priority high; alter table OBIEE_COUNTY_HIER inmemory priority high; alter table OBIEE_GEO_AIRPORT_BRIDGE inmemory priority high; alter table OBIEE_GEO_ORIG inmemory priority high; alter table OBIEE_ROUTE inmemory priority high; alter table OBIEE_TIME_DAY_D inmemory priority high; alter table OBIEE_TIME_MTH_D inmemory priority high; alter table ONTIME_DELAY_GROUPS inmemory priority high; alter table PERFORMANCE inmemory priority high; alter table PERFORMANCE_ENDECA_MV inmemory priority high; alter table ROUTES_FOR_LINKS inmemory priority high; alter table SCHEDULES inmemory priority high; alter table SERVICE_CLASS inmemory priority high; alter table UNIQUE_CARRIERS inmemory priority high;

SQL> @display_table_inmem_details.sql SQL> select table_name 2 , inmemory 3 , inmemory_priority 4 from user_tables 5 / TABLE_NAME INMEMORY INMEMORY -------------------- -------- -------- AIRCRAFT_GROUP ENABLED HIGH UNIQUE_CARRIERS ENABLED HIGH SERVICE_CLASS ENABLED HIGH SCHEDULES ENABLED HIGH ROUTES_FOR_LINKS ENABLED HIGH PERFORMANCE ENABLED HIGH ONTIME_DELAY_GROUPS ENABLED HIGH OBIEE_TIME_MTH_D ENABLED HIGH OBIEE_TIME_DAY_D ENABLED HIGH OBIEE_ROUTE ENABLED HIGH OBIEE_GEO_ORIG ENABLED HIGH OBIEE_COUNTY_HIER ENABLED HIGH DOMESTIC_SEGMENT ENABLED HIGH DISTANCE_GROUP_250 ENABLED HIGH DEPARBLK ENABLED HIGH CARRIER_REGION ENABLED HIGH …

1 2



Query Execution Plans Show In-Memory Feature Used

Predicate Information (identified by operation id): --------------------------------------------------- 7 - access("D1"."C3"="D2"."C4" AND SYS_OP_MAP_NONNULL("D1"."C4")=SYS_OP_MAP_NONNULL("D2"."C3")) 13 - access("T233484"."STCTY_FIPS"="T233820"."SC_CODE") 15 - access("T233484"."AIRPORT"="T233609"."ORIGIN") 17 - access("T233732"."CAL_MONTH"=TO_NUMBER("T233609"."MONTH") AND "T233732"."CAL_QTR"=TO_NUMBER("T233609"."QUARTER") AND "T233732"."CAL_YEAR"=TO_NUMBER("T233609"."YEAR")) 18 - inmemory("T233732"."CAL_MONTH">=6 AND "T233732"."CAL_MONTH"<=12) filter("T233732"."CAL_MONTH">=6 AND "T233732"."CAL_MONTH"<=12) 19 - inmemory(TO_NUMBER("T233609"."MONTH")>=6 AND TO_NUMBER("T233609"."MONTH")<=12) filter(TO_NUMBER("T233609"."MONTH")>=6 AND TO_NUMBER("T233609"."MONTH")<=12) 21 - inmemory(SYS_OP_BLOOM_FILTER(:BF0001,"T233484"."AIRPORT")) filter(SYS_OP_BLOOM_FILTER(:BF0001,"T233484"."AIRPORT"))

|* 15 | HASH JOIN | | 136 | 11288 | | 142 (6)| 00:00:01 | Q1,03 | SCWC | | 16 | JOIN FILTER CREATE | :BF0001 | 136 | 4896 | | 142 (6)| 00:00:01 | Q1,03 | PCWP | |* 17 | HASH JOIN | | 136 | 4896 | | 142 (6)| 00:00:01 | Q1,03 | SCWC | | |* 18 | TABLE ACCESS INMEMORY FULL | OBIEE_TIME_MTH_D | 161 | 1610 | | 1 (0)| 00:00:01 | Q1,03 | SCWP | |* 19 | TABLE ACCESS INMEMORY FULL | DOMESTIC_SEGMENT | 771 | 20046 | | 141 (6)| 00:00:01 | Q1,03 | SCWP | | 20 | JOIN FILTER USE | :BF0001 | 1787 | 83989 | | 1 (0)| 00:00:01 | Q1,03 | PCWP | | |* 21 | TABLE ACCESS INMEMORY FULL | OBIEE_GEO_AIRPORT_BRIDGE | 1787 | 83989 | | 1 (0)| 00:00:01 | Q1,03 | SCWP | | 22 | JOIN FILTER USE | :BF0000 | 3221 | 141K| | 1 (0)| 00:00:01 | Q1,03 | PCWP | | |* 23 | TABLE ACCESS INMEMORY FULL | OBIEE_COUNTY_HIER | 3221 | 141K| | 1 (0)| 00:00:01 | Q1,03 | SCWP |



Oracle Database 12c In-Memory, No Aggs, Response Time

•Better than 11gR2 row-store on-disk response times, but not as fast as 11gR2 with Aggs •But doesn’t 12c In-Memoryhelp with aggregation too?

0"

10"

20"

30"

40"

50"

60"



Delay"Increase"A>er"

Depart"

Delay"increase"A>er"

Dept"Unprompt"

Delay"Performance"

Matrix"

Time"Blocks"Delay"



Pie"


11gR2"DB,"No"Aggs"


12cR1"DB,"No"Aggs"



In-Memory Aggregation with In-Memory Option

•For suitable queries, In-Memory Option creates a temporary in-memory cube for aggregation • “Vector group by” optimiser transformation

•Used when query aggregates large amount of rowsafter filters applied, and joinsto multiple (4+) dimensions

•Uses vector access to cubedata similar to approach usedby Oracle OLAP •“Spiritual successor toOracle OLAP functionality”



Adding Aggregate Tables to 12c In-Memory Dataset

•Maximum benefit though is achieved by adding summary tables to 12c In-Memory dataset •Combining fast column-store access to detail-level data + precomputed aggregates provides optimal database solution for Airline Delays application

•All queries now have <1s response time, consistently

0"

10"

20"

30"

40"

50"

60"



Delay"Increase"A>er"Depart"

Delay"increase"A>er"Dept"Unprompt"

Delay"Performance"

Matrix"

Time"Blocks"Delay"


Delay"Late"Flight"Cause"Pie"


11gR2"DB,"No"Aggs"


12cR1"DB,"No"Aggs"

InN12cR1"DB"with"Agg"Tables"



Adding Exalytics and In-Memory Analytics to OBIEE 11gMark Rittman, CTO, Rittman Mead February 2015



No Silver Bullets : OBIEE Performance in the Real World Mark Rittman, CTO, Rittman Mead Oracle University Expert SummitDubai, February 2015

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com

OBIEE Performance in the Real World

•Do It Right, First Time ‣What Makes A Performant OBIEE System?

• If It’s Not Done Right, Know How To Figure Out What Is Broke ‣Practical elements of diagnosing the causes of performance issues ‣Methodical analysis - “end to end”


The Requirement


The Problem

•Lots of moving parts and complexity •Overall solution crosses disciplines and job roles ‣OBIEE developer ‣Database developer ‣DBA ‣Server Admin ‣Network ‣SAN ‣LDAP ‣etc


The Additional Problem


Performance Beyond the “Best Practices”

•Tear down the reliance on “Best Practice”, but with a viable, better, alternative instead.


The Only Answer

EVIDENCE-BASED DESIGN AND DIAGNOSTICS


Time Profile

•To understand why is it slow, we first must understand where is it slow

•Approach championed by Cary Millsap / Method-R • “Thinking Clearly About Performance” (2010)

Performance improvement is proportional to how much a program uses the thing you improved.

— Amdahl’s Law


The OBIEE stack

Request

Response

BI Server

BI Server

Presentation Services

Managed Server

Managed Server

JavaHost

Presentation Services

Web Browser

Web Browser

BI Plug-in

BI Plug-in

DWH

[ ... ]

[ ... ]

DWH

[ ... ]

[ ... ]

User WebLogic Server OBIEE system components Data Source(s)

Net

wor

k

Net

wor

k

Not all components listed, eg cluster controller

-------------------- SQL Request, logical request hash: 810510fe SET VARIABLE QUERY_SRC_CD='Report',SAW_SRC_PATH='/shared/Test/test1 jan\/feb';SELECT 0 s_0, "Sales - Fact Sales"."Dim Products"."Product Type" s_1, "Sales - Fact Sales"."Dim Times"."Month" s_2, SORTKEY("Sales - Fact Sales"."Dim Times"."Month") s_3, "Sales - Fact Sales"."Fact Sales"."Cost" s_4 FROM "Sales - Fact Sales"

C1 C2 C3 C4 ---------- -------------------- -------------------- ---------- 4636.93 Bread Assortments FEB-2000 200002 4583.53 Bread Assortments JAN-2000 200001 3445.16 Bread Clubs FEB-2000 200002 3542.49 Bread Clubs JAN-2000 200001 4735.63 Cold Drinks FEB-2000 200002 4565.21 Cold Drinks JAN-2000 200001 4604.19 Gifts & Baskets FEB-2000 200002 […]

------------------------------------------------------------------------------------------- s_0 s_1 s_2 s_3 s_4 ------------------------------------------------------------------------------------------- 0 Bread Assortments JAN-2000 200001 4583.53 0 Bread Clubs JAN-2000 200001 3542.49 0 Cold Drinks JAN-2000 200001 4565.21 0 Gifts & Baskets JAN-2000 200001 4608.01 0 Hot Drinks JAN-2000 200001 5768.62 […]

Sending query to database named orcl (id: connection pool named Connection Pool 01

select sum(T117.FCAST_COST_AMT) as c1, T107.PROD_TYPE_DESC as c2, T127.MONTH_DESC as c3, T127.MONTH_YYYYMM as c4 from GCBC_SALES.TIMES T127 /* Dim_TIMES */ , GCBC_SALES.PRODUCTS T107 /* Dim_PRODUCTS */ , GCBC_SALES.SALES T117 /* Fact_SALES */


Time Profile in Action

•End-user reports a response time of 40 seconds. •How can we make it faster?

‣Where did the time get spent?

Response

BI ServerManaged Server

Presentation ServicesWeb Browser BI Plug-in

User WebLogic Server OBIEE system components

DB Query 1 25 seconds

DB Query 2 5 seconds

Data Processing 10 secondsPage Generation

5 seconds

DWH

Data Source(s)

DWH

Tim

e, s

econ

ds

Component


Time Profile in Action

Step Action Response Time (s) %

1 Physical SQL 1 execute on DB 25.00 62.5

2 Physical SQL 2 execute on DB [5.00] —3 BI Server does work on DB results 10.00 25.0

4 Presentation Services generates page 5.00 12.5

Total 40.00 100.0

DB Query 1

DB Query 2

Data Processing

Page Generation

Time profile shows clearly : 1.Improve performance of Query 1 2.Push work into single query


Deep Dive into BI Server Time from nqquery.log

Elapsed

Response

Physical

Physical Query x

Physical Query 1 BI Server processing (federation, calculation, aggregation)

Send to client, wait for acknowledgement

Eg. Presentation Services generating pivot tables, table scrolling/paging, etc.

Com

pilat

ion

DB C

onne

ct

Logical Query Summary Stats: [...] Compilation time c (seconds)

Physical query response time p1 (seconds)

Physical query response time px (seconds)

Logical Query Summary Stats: Elapsed time e

Logical Query Summary Stats: [...] Response time r

Physical Query Summary Stats: [...] DB-connect time d (seconds) r - d - c - max(p) e - r

http://ritt.md/nqquery


Usage Tracking’s blind spot

•Usage Tracking is good, but it omits the Total Elapsed Time

•This can hide problems:

•Usage Tracking is great, but do be aware of this potential limitation to your visibility of where time has gone.

Elapsed

ResponseSend to client, wait for acknowledgement

nqquery.log: Logical Query Summary Stats: [...] Response time r Usage Tracking: S_NQ_ACCT.TOTAL_TIME_SEC

nqquery.log: Logical Query Summary Stats: Elapsed time e Usage Tracking: [Not Available]

uh oh!


Request

Response

Iterative Approach

•Be sure to account for all time, end-to-end (nose to tail) •Expand time profile to focus on where the time is going in particular • In reality, time profile based on BI Server alone will help in a lot of cases

BI ServerManaged Server


User WebLogic Server OBIEE system components

DWH

Data Source(s)

DWH

Net

wor

k

Net

wor

k

nqquery.logsawlog.log EM Active Reports, V$SQL, etcBrowser tools WLS logs


Digging Deeper with DMS Metrics

•OBIEE supports Fusion Middleware’s Dynamic Monitoring System (DMS)

•Once we have found WHERE the time has gone, DMS metrics are one way to help us find out WHY

•Hundreds of low-level metrics, ranging from the obvious (BI Server Total Sessions) to the less obvious (Peak_DXEParameterizedImpl_Count)


Accessing DMS Metrics

•DMS Spy •opmn •EM FMC •EM12c •obi-metrics-agent + graphite + grafana

http://ritt.md/oma-intro



Resp

onse

tim

e

Number of concurrent users

•Performance: Response time is slow for one user •Capacity: Response time degrades as user concurrency increases

•Fix Performance problems at root cause; adding capacity alone is generally not sensible •Improve the performance of an underlying Capacity problem and you might offset the need to add any additional capacity at all.

Is your problem Performance or Capacity?Ba

dG

ood


Performance Diagnostics Approach

•Performance ‣Build a time profile ‣Don’t know where to start? Try Usage Tracking or nqquery.log

-Long running queries -Logical queries returning lots of data to the client -Logical queries returning lots of data from the DB but small % to the client -Logical queries requiring lots of physical queries

•Capacity ‣Examine capacity metrics (OS, DMS) over time ‣Correlate with reported problems

Queuing DMS metrics: Oracle BI DB Connection Pool/* -> Current Queued Requests Oracle BI PS Chart Engine -> Current Charts Queued Oracle BI PS Query Cache -> Current Queued Queries Oracle BI PS Thread Pools/* -> Current Jobs Queued Oracle BI Thread Pool/* -> Current Queued Requests


So What Does “Good” Look Like?


This Is Not “Best Practice”

•There is no one right way : It Depends

The only “best practice” you should be using all the time is “Use Your Brain”. — Steven Robbins / Tom Kyte


The Bucket List of Performance Gains

★Don’t Do It

★Do It Only Once

★Do It Less Often

★Do It More Efficiently

(h/t Greg Rahn)

https://twitter.com/gregrahn/status/423685471776935936


‘Database Pushdown’ (Do It More Efficiently)

•Reduce the amount of work/data handled further up the stack •Federation across sources can be a challenge ‣c.f. BI Server caching

BI Server

Managed Server


DWH

[ ... ]

User WebLogic Server OBIEE system components Data Source(s)

Net

wor

k

Net

wor

k


Feeding the Excel Monster (Don’t Do It/Do It More Efficiently)

•What’s being done with the data once it’s in Excel? ‣Could it be done in OBIEE instead?

•Alternatives to Export from Analysis/Dashboard: ‣Oracle have specific recommendations (DocID 1558070.1 p.13) -Favour CSV export over Excel -Favour BI Publisher export over OBIEE Analysis Export

‣Use Logical SQL against BI Server’s ODBC/JDBC interface directly ‣Dump direct from the database


Don’t Do It / Do It Less Often

• “Filter Early” •Table Prompt != Filter

•Report by Exception

•Make sure Dashboards have default/mandatory prompts(http://ritt.md/obi-prompts)

Yes, please do….

•Be smart about Dashboard and Analysis design - don’t cram everything into one page •Less clutter : Better user experience & Better performance

http://ritt.md/obi-prompts


Not forgetting …

•Do It More Efficiently ‣Aggregation

•Do It Less Often ‣BI Server caching

•Database optimisation (partitioning, indexing, parallelism, statistics, etc)

•Balanced Hardware configuration


OBIEE Performance in the Real World

•Do It Right, First Time

•Methodical analysis - “end to end”

EVIDENCE-BASED DESIGN AND DIAGNOSTICS


References & Further Reading

•Cary Millsap - “Thinking Clearly About Performance” ‣http://queue.acm.org/detail.cfm?id=1854041

•Blog series: OBIEE and Performance ‣http://ritt.md/obiee-performance

•obi-metrics-agent ‣http://ritt.md/oma-intro

•Greg Rahn - “The Core Performance Fundamentals Of Oracle Data Warehousing – Balanced Hardware Configuration” ‣http://wp.me/p3cJT-by

•Oracle documentation - “Oracle® Database 2 Day + Data Warehousing Guide - Balanced Hardware Configuration” ‣E25555-03

http://queue.acm.org/detail.cfm?id=1854041

http://ritt.md/obiee-performance


http://wp.me/p3cJT-by



No Silver Bullets : OBIEE Performance in the Real World Mark Rittman, CTO, Rittman Mead Oracle University Expert SummitDubai, February 2015



Source Control & Concurrent Development for OBIEE Mark Rittman, CTO, Rittman Mead Oracle University Expert SummitDubai, February 2015


OBIEE Software Development Life Cycle (SDLC)

•A successful OBIEE implementation is more than just data modelling and analytics.

•Software Development LifeCycle (SDLC) is of great importance to all involved in OBIEE: ‣Developers ‣Release Managers ‣Testing ‣Project & Delivery Management

•Two key areas of importance: ‣Source Control ‣Concurrent Development

[email protected] www.rittmanmead.com @rittmanmead

Source Control in OBIEE

(Source Control == Revision Control == Version Control)

118


http://www.rittmanmead.com

http://twitter.com/rittmanmead


Without Source Control We Have Chaos!


Why Do We Need Source Control?

•Every mature OBIEE development should be using Source Control!

‣Reduce risk when deploying

‣Improve ease of maintenance and development

‣Pre-requisite to concurrent development method (discussed later)



Who changed it?

What did they change?

When did they change it?


•Label (tag) points in the code line

‣Mark code bundles for release to a given environment -Audit what went where when

‣Revert to “known-good” point



Isn’t Source Control Scary and Complicated?

•No!

•Well… maybe a bit

•But you don’t really have a choice


Without Source Control We Have Chaos!


Source Control is Mandatory

•Any mature OBIEE development should be using Source Control

•One developer or one hundred, still relevant


What Are the Tools for Source Control in OBIEE?

•Familiarity with the tool and organisational acceptance is key

•Two common ones (but plenty of others) ‣git ‣Subversion (SVN)

• If you are wanting to do concurrent development, git is excellent ‣Modern technology ‣Distributed


Just Use Source Control Already


OBIEE Artefacts in Source Control

•OBIEE primarily stores its objects in binary form ‣We’ll talk about MDS XML later

•Source Control tools and their merge workflows are designed around plaintext objects

•Therefore Source Control becomes a snapshot / point-in-time capture of your OBIEE environment

RPDPresentation CatalogSecurityComponent configurationCustom skin/styleETL, DB schemas, etc

[email protected] www.rittmanmead.com @rittmanmead

Concurrent RPD Development in OBIEE

129



http://twitter.com/rittmanmead


The RPD

•At the core of OBIEE is the RPD - the brains of the operation!

•A metadata model that generates SQL requests at runtime from user report requests

•Most concurrent OBIEE development problems centre around the RPD


The Problem With Serial RPD Development

• It doesn't scale well

•Releases are complicated and risky

Test ProdDev

MultipleDevelopers


Feature Driven Development

•Development is broken up into “Features”, such as: •New logical column •Add a dimension •Fix a bug

•Great for flexible (Agile/agile) approach to delivery •Features only released when ready •Features don't block others •Selectively release features


OBIEE Development Nirvana

•RPD development rate that scales with your team

•Feature-Driven Development

•Full source control integration

•Because of the necessary framework, makes Continuous Integration very possible


The Harsh Reality

•OBIEE does not natively support concurrent development integrated with source control ‣And it never will until Oracle re-engineer OBIEE and/or the RPD storage format

•All options require a defined process to be followed and associated training for staff •There is no perfect software-based solution that will force ‘bad’ developers to be good

•You will always need a “Dungeon Master” to oversee and track developments, manage conflicts, and support developers.


The Harsh Reality

•There are four options to consider for concurrent development in OBIEE ‣Online editing ‣MUD ‣Source control + MDS XML RPD ‣Source control + Binary RPD

•They all suck! •This is 2015, not 1995!

•Which sucks the least?… ‣What can we do to mitigate the suckiness?


Online RPD edits

•Zero setup required

•RPD updated automatically - no redeployment or restarts needed

•Concurrent development limited by high level locks taken when objects checked out

•Everyone's favourite error : "Transactional update failed”

Developer #1

Developer #2

1. Online edit 2. MUD 3. Source control + MDS XML RPD 4. Source control + Binary RPD


Online RPD edits

•No audit of changes made

•No native integration with source control •No tagging or packaging for releases •No rollback of changes / restore to point in time

•Conclusion : ‣Good for individual developers in isolation making sandbox changes ‣Useless anywhere else in the development cycle process

1. Online edit 2. MUD 3. Source control + MDS XML RPD 4. Source control + Binary RPD


Multi User Development Environment (MUD)

•Administrator divides main repository into projects; self-contained RPD subsets

•Master repository is then published to a network share

•Projects are worked on independently, and then merged back into the master RPD

•Uses the repository compare and merge features under the covers

OBIEE Sandbox Environment

Developer #1

OBIEE Sandbox Environment

Developer #2

MUD Administrator

Master Repository

SubsetRPD

SubsetRPD

1. Online edit

2. MUD 3. Source control + MDS XML RPD 4. Source control + Binary RPD


Multi User Development Environment (MUD)

•Closest thing there is to native concurrent development functionality in OBIEE

•Onus and power for conflict resolution is with the developer, not source master

•Known bugs (eg Variables, parent-child tables)

•Doesn’t easily support feature-driven development

•No native integration with source control •No tagging or packaging for releases •No rollback of changes / restore to point in time

1. Online edit

2. MUD 3. Source control + MDS XML RPD 4. Source control + Binary RPD


We’ve Got To Have Source Control!

•Any project should always be using source control •To do properly flexible concurrent development, we need to take a feature-driven approach

➡Thus we use branch based source control

•But - this requires the ability to merge files within the source control tool from multiple branches into one

develop

feature/RM-x

feature/RM-y


Source Control Merging

•When branches are combined in source control, the tool will automagically merge files and folders

•Source control tools can only automatically merge files that are not binary…


Is MDS XML the answer?

•The RPD is broken up into a hierarchical set of XML files representing object types such as Logical Tables, Logical Table Sources, etc

•Because there are multiple plain text files, can the source control tool can carry out the merge itself?


Source control + MDS XML RPD

•Package up all artefacts in Source Control, providing release and rollback capabilities

•Store the RPD in MDS XML, a plaintext alternative to the binary RPD format

•Deceptively alluring option for concurrent development:

1. An RPD stored in MDS XML is plain text 2. git can merge code that is plain text from

multiple branches 3. Let's merge MDS XML with git! 4. Uh oh….

1. Online edit 2. MUD

3. Source control + MDS XML RPD 4. Source control + Binary RPD


Source control + MDS XML RPD - the good

‣It works in simple and specific use-cases

Branch A

Branch B

Merged RPD

Dim Stores

Dim Staff





‣Separate commits in git





‣Happily merged!

Dim Stores

Dim Staff




Source control + MDS XML RPD - the bad

•Even though MDS XML is not binary, it is still a structured file format containing with application logic ‣Only the Administration Tool understands the RPD object model

•Source control tools are not intelligent enough to be able to parse it to understand conflicts




Source control + MDS XML RPD - the bad

•Let’s add new columns to same table in separate branches

•Should be simple - there’s no logical conflict

Branch A Branch B





‣Source control tool looks at the two versions of MDS XML file as dumb plaintext, and throws a conflict

‣User now has to manually fix the XML, which is tricky and error prone ‣It’s also unnecessary - merging through the Administration Tool would be automatic




Source control + MDS XML RPD - the ugly

‣MDS XML files are named based on internal GUIDs (mdsid) ‣Same logical object can have different mdsid, and so source control sees it as unique ‣Allowing source control tool to merge files can create a corrupt RPD ‣For example, an identical object created in multiple branches

Branch A Branch B




Source control + MDS XML RPD - the ugly

‣Same object gets a different mdsid, thus a different filename and so gets duplicated in the resulting merge, because source control doesn’t see it as the same object

‣Result: dodgy RPD, user cannot edit the SALES table.

‣Exactly what the Equalize function was designed for in the AdminTool merge!

The same object definition!




Source control + MDS XML RPD - Summary

•You cannot reliably merge the RPD except through Oracle’s own tools ‣MDS XML, whilst plaintext, is still structured

-Unnecessary conflicts when merging -Potential inconsistencies / corruption

‣MDS XML uses GUIDs to name files, so source control mayinadvertently duplicate objects without notification -RPD problems may not always be immediately apparent

•MDS XML merging may work for simple isolated changes to individual objects; as a development strategy this isrestrictive.




Source control + Binary RPD

•Package up all artefacts in Source Control, providing release and rollback capabilities

•Use the common git-flow branching method, with development divided up into features ‣Provides clear structure and process for managing branches and releases

‣RPD stored as binary. Three-way merge with OBIEE tools done where necessary.

develop

feature/RM-x

feature/RM-x

Original

Current

Modified

1. Online edit 2. MUD 3. Source control + MDS XML RPD

4. Source control + Binary RPD

‘Secret Sauce’ : merge.bat script



•Conflict resolution script: ‣Uses git-flow to determine 3-way merge candidates ‣Attempts automatic merge (comparerpd/patchrpd) ‣Launches Administration Tool for user interaction if necessary

1. Online edit 2. MUD 3. Source control + MDS XML RPD

4. Source control + Binary RPD


So which methods sucks the least?

Multiple developers

Integration with source control

Feature driven development

Ease of use

Overall


Yes Yes Yes Medium Not perfect, but the best option

MUD Yes Manual No Medium Could be worse


Yes Yes Only if objects fully isolated

Medium Requires too much process to mitigate against RPD corruption and unnecessary merge conflicts Unnecessary conflict merges.Online

editing Not really No No Easy Not viable


The Harsh Reality

•OBIEE does not natively support concurrent development integrated with source control ‣And it never will until Oracle re-engineer OBIEE and/or the RPD storage format

•All options require a defined process to be followed and associated training for staff •There is no perfect software-based solution that will force ‘bad’ developers to be good

•You will always need a “Dungeon Master” to oversee and track developments, manage conflicts, and support developers.


Checking It All Still Works

•For concurrent development to be successful, you need automated testing

Before

After

Test Result: Fail


Summary

• If you're not doing source control, go and do it. ‣Now. ‣Even for serial development ‣Even if you’re the only developer

• If you are doing source control, then you can use it to support concurrent development ‣But don't go anywhere near concurrent development until you've sorted source control

•The only way to do safe and reliable concurrent development is using the binary RPD stored in source control such as git



Source Control & Concurrent Development for OBIEE Mark Rittman, CTO, Rittman Mead Oracle University Expert SummitDubai, February 2015



Making the Move to BI Apps 11gMark Rittman, CTO, Rittman Mead Oracle University Expert SummitDubai, February 2015



What are the Oracle BI Applications?

•Pre-built and pre-integrated horizontal and industry-specific applications built around OBIEE •Provides pre-built ETL, conformed dimensional data warehouse, RPD and Catalog •Covers Oracle EBS, Peoplesoft, Siebel, SAP and Fusion Applications

IntegratedAnalytic Apps

•  Financials •  HR •  Procure & Spend •  Projects •  Supply Chain •  Order Management

•  Sales •  Marketing •  Service •  Contact Center •  Price •  Loyalty •  Hyperion Planning

•  Hyperion Close •  Budgeting & Forecasting •  Profitability Management

•  Communications •  Healthcare •  Retail •  Financial Services

Run 5-10X Faster Leverage Built-In

Best Practices



Oracle BI Applications Packaging

•BI Apps is sold as a series of analytic applications - Financial Analytics, HR Analytics etc •Vertical apps for pharma, banking etc also available •OBIEE, Oracle Database and ETL tool have to be licensed separately •Customers typically license 3-4 applications, then build out from there



Pre-Defined ETL Routines from EBS, PSFT, Siebel, JDE, SAP

• Integrated, conformed dimensional data warehouse •Deployable on Oracle, MS SQL, IBM DB/2 and Teradata •Uses embedded ETL tool (historically Informatica) to load data from sources to DW •Staging tables and presentation tables •Allows modular deployment •Lowest grain of information •Prebuilt aggregates •History tracking • Indexing



Oracle BI Apps 7.9.x and Earlier

•Historically, BI Apps has used Informatica PowerCenter as the embedded ETL tools ‣Controlled and orchestrated by DAC (Data Warehouse Administration Console)

• “Loosely-coupled” integration that can be complex to set up, but otherwise works well •BI Apps ships with pre-defined INFA mappings, which are then called as tasks within execution plans

•This option will continue - but now there is also the option to use ODI 11g



Introducing Oracle BI Applications 11g and ODI 11g

•All-new, 11g release of the BI Applications •New content, applications, uptakes OBIEE 11g visuals etc •Simplified topology - now uses ODI 11g for the ETL ‣Informatica support coming soon

•No DAC - uses web-based configuration tools, and ODI, to control and orchestrate the ETL

•Lower TCO, faster ETL, simpler architecture



New Content in BI Apps 11.1.1.7.1

•Two new analytic applications: ‣Student Information Analytics ‣Indirect Spend Analytics

•New content for existing applications ‣Financial Analytics ‣HR Analytics ‣Project Analytics etc

•Dashboards and analyses updated for 11g ‣Performance tiles ‣Trellis charts ‣Hierarchical columns etc



BI Apps 11.1.1.7.1 Source System Support

* As of Jan 2013, from Oracle Product Roadmap presentation. Subject to change.



New in BI Apps 11.1.1.8.1 - Additional Content



New in BI Apps 11.1.1.8.1 - Data Lineage / Impact Analysis

•Trace data lineage from ERP/CRM source to OBIEE dashboard •Understand impact of source system / mapping / DW changes •End-to-end metadata analysis fromsource to report



BI Apps 11.1.1.8.1 Source System Support



BI Applications Product Roadmap



Migrating and Upgrading to BI Apps 11g

•BI Apps 11g is ODI-only, no plans to support INFA as ETL tool ‣7.9.6.x version will not receive any new content, only bug fixes etc ‣INFA-based customers will need to migrate to ODI + BI Apps 11g to get new content

•No immediate pressure to migrate, but advisable if new content etc required over time



7.9.6.4 vs. 11g

•So how does one choose between 7.9.6.x and 11g for new implementations? •At the moment, choice is driven by ETL tool preference ‣If you want Informatica - go with 7.9.6.x (but no further development) ‣If you want ODI - go with 11.1.1.7.x

•So then more down to known-and-tested vs. new-and-more-features •Going forward, 11g should support all sources, and both ETL tools ‣So more down to ETL tool preference - though no easy way to switch later on ‣And ODI will be Oracle development focus - Informatica mappings will be ports

• In vast majority of cases - go with OBIA 11g and ODI11g



Oracle BI Applications 11g Product Architecture

•All server-side BI Apps components now run withinWebLogic Server Domain

•Web-based apps for configuring and controlling the ETL process

•ODI agent runs within WebLogic domain, uses WLS security etc

•ODI Console embedded in web apps, used for web-based monitoring of loads

•ODI Studio available for developer tasks •No DAC - functionality now handled by ODI,Configuration Manager and FSM

•GoldenGate for trickle-feed loadinginto staging area / ODS



Taking a Closer Look...

1.Installation process 2.New Administration Tools 3.Post-Install Configuration 4.How ODI11g replaces Informatica, and the Informatica Repository 5.Creating and Executing Data Loads



Installation Steps

1. Use RCU to create OBIEE repository tables 2. Install OBIEE 11g using Enterprise Install type 3. Run OBIA RCU to create DW schema, BIA schema,

ODI repository etc 4. Install OBIA 11g incl. ODI into

OBIEE middleware home 5. Apply OBIA patches 6. Use Configuration Utility to extend WLS domain

to include OBIA applications 7. Upgrade WLS to 10.3.6 8. Run post-install config scripts 9. Configure ODI Studio to use WLS security



Configuration Manager, and Functional Setup Manager

•Functionality of the DAC has been passed to three products ‣ODI - for details on mappings, dependencies etc ‣Configuration Manager - for system-wide configuration ‣Functional Setup Manager - for app-specific configuration

•Deep integration between the tools •Faster deployment and configuration



Use of ODI 11g as the Embedded ETL Engine

•ODI, rather than Informatica, is used as the embedded ETL engine • Individual mappings are grouped into scenarios, equivalent of workflows in Informatica

•Scenarios are then orchestrated into load plans, the equivalent to execution plans in the DAC

•ODI uses the target DW (Oracle initially) for ETL ‣Actually “ELT”, for extract-load-transform

•ODI agent controls the process - equivalent to DAC server

•Configuration metadata is held in the ODI repository, and in repository tables used by CM and FSM



Leveraging Core ODI Functionality

•Oracle Data Integrator 11g is used as the embedded data loading tool for the BI Apps •Uses core ODI functionality - interfaces, scenarios, load plans, topologies, agents etc •ODI Studio can be used to view ETL internals, or just manage through web-based tools •Uses ODI versioning for customization, ODI release mgmt etc



Inside the ODI BI Apps Repository

•BI Apps “Adapters” become Projects withinthe ODI work repository

• INFA workflows / DAC tasks becomescenarios, mapping to one or more interfaces

•Data source connections stored in ODI masterrepository, viewed in Topology navigator

•Execution plans become load plans •ODI versioning used when customizations are made, instead of creating new adapters

•Groupings of facts and dimensions are definedin Configuration Manager, which has its ownrepository tables (BIACM)



Initial Configuration, Data Sources and Domain Value Load

•Several steps need to be performed before a full data load can take place 1.Select products to enable 2.Define source systems (EBS etc) 3.Create data source connections 4.Load domain values (account codes etc) from source into DW

•Performed using Config. Manager and FSM •All settings automatically pushed-through to ODI repository ‣JDBC connection details to ODI repository ‣File locations, etc.



Fact Groups - Equivalent to DAC Subject Areas

•BI Apps 7.9.x had the concept of subject areas and execution plans, within the DAC •Allowed groups of facts + dimensions to be selected, and then loaded in the correct sequence

•Handled dependencies, running steps in parallel, restarting etc

• In BI Apps 11.1.1.7.1, this is now handled through fact groups, and load plans

•A set of fact tables for a particular subject area •Link to their dimensions using FKs •Load plans are generated on-demand (equivalent to execution plan build step), and sequence ODI scenarios that are linked to the fact tables in the ODI metadata



Load Plans - Equivalent to DAC Execution Plans

•Load plans were introduced with ODI 11.1.1.5, and are a way of orchestrating sets of compiled ETL steps

•Steps can be run in parallel or sequence •Exceptions can be defined, to execute steps on ETL failure

•Load plans can be restarted, skipping previously completed steps

•Can run individual compiled interfaces, orscenarios (equivalent to INFA workflows)

•So how does ODI generate the particular load plan required for a set of arbitrarily-selected fact groups?



The Load Plan Generator

•Load Plan Generator (LPG) is a JEE library installed into the WebLogic Domain alongside OBIEE, CM etc

•Automatically generate the optimal load plan for a set of fact groups •Called from the Configuration Manager web interface, or by developers as a plug-in to ODI Studio

•Uses fact > dimension FK relationships, and then links back to staging tables and temp tables, to create the required load plan



Automatic Population of Domain Values

•Uses special type of load plan called a “Domain-Only Extract and Load (SDE and SIL)” •Retrieves flexfield and other values from EBS, PSFT etc and automatically populates BIA metadata tables - no need to manually complete CSV files

12

3



Initiating, and Viewing the Status of, Load Plans

•BI Apps load plans are generated, and then initiated, from Configuration Manager •Once initiated, status can be monitored either in CM, or in ODI Studio •View ETL failure reasons, see error codes, restart from point of error

1

2

3



Performing a Full Data Load Step 1 : Create Load Plan

•Similar to creating a domain values load plan - select fact groups, generate new load plan •Relies on “offerings” (applications) being enabled at an earlier stage •First stage is to generate this load plan, like the “build” phase for DAC execution phases •But no need to get involved with parameters etc - all handled behind the scenes



Performing a Full Data Load Step 2 : Execute Plan

•New load plan executed through Configuration Manager (or to a schedule) •Creates new instance of the load plan within ODI repository •View progress of load plan in CM, in ODI Console, or ODI Studio



Performing a Full Data Load Part 3 : Deal with any ETL Issues

•Data loads can fail for a number of reasons ‣New data that causes key constraints, data out of range, duplicates ‣Out of tablespace datafile etc

•ODI load plans are restartable, can raise exceptions, showdebug information from within ODI Studio Operator ‣Less log files to look through ‣Obvious to ODI developers what the issue is ‣Straightforward way to write exception handling code,to for example extend datafiles, raise alerts etc

•Once issue addressed, just restart load plan ‣Automatically resumes from failed step



BI Apps 11.1.1.7.1 and GoldenGate

•Oracle’s cross-platform data replication (CDC) solution •Near-zero impact of ongoing data extraction ‣No ETL batch load windows required ‣Full reload without touching source

•Consistent CDC approach regardless of source •Potentially reduced ETL runtimes ‣Transforms and loads all performed locally ‣Near real-time ETL possible, & 24x7 deployments ‣Great enabler for cloud deployments

•Trickle-feed of data updates over WAN



How Does GoldenGate Integration Work?

•Creates new data layer called “Source-Dependent Staging” ‣Replica of source tables, held locally to BI Apps and with GG/ODI journaling

•GoldenGate replicates OLTP data into this SDS schema in real-time •ODI / BI Apps install creates ODI procedures for creating SDS DDL, based on OLTP schema •Pre-build ODI procedures for initial load,and then GG-based incremental loads ‣Uses custom procedure rather than JKM

•Possibility of amending GG routines totransform, project and filter OLTP data? ‣Could be a way to create an ODS, fedfrom multiple GG OLTP sources?



New in BI Apps 11.1.1.8.1 - Endeca Information Discovery

•Oracle Endeca Information Discovery adds “data discovery” + search capabilities to BI Apps • In-memory click-and-filter analysis of large attribute-based datasets •Search, parsing, lexical analysis capabilities over text and descriptive fields

•Extends BI Apps capabilities and integrates with ODI load plans



OEID Data Loads Integrate with BI Apps ODI Load Plans

•Uses new OEID Plug-in for ODI11g - loads OEID data domain via BI Apps RPD



Sample OEID Studio Applications for BI Apps Subject Areas



New in BI Apps 11.1.1.8.1 - Data Lineage / Impact Analysis

•BI Apps 11g systems have multiple levels of metadata •Data in reports can be traced through the load process,BI metadata layer, underlying DW and back to the source DB

• Important to understand data lineage to be certain that datais source correctly, understand changes / transforms

• Impact analysis helps us understand what effect changeswe make to sources and transformations will have on reports and dashboards

•New in 11.1.1.8.1 - data lineage and impact analysis

OLTP Tables / Columns

SDE Interfaces

SIL Interfaces

DW Tables

BMM Layer Mappings

Presentation Tables / Columns

Dashboards / Reports



Automatic Data Lineage Load Routines

•Takes exports of RPD and catalog saved as files •Combines with data extracts from BIACM and ODI repositories •Loads using an ODI load plan •Rerun to reload data lineage tables •Stored in DW database schema



Data Lineage and Impact Analysis Dashboards

•RPD subject area and catalog objects provided as starter •Use data lineage subject areas to create own analyses



Making the Move to BI Apps 11gMark Rittman, CTO, Rittman Mead Oracle University Expert SummitDubai, February 2015



Oracle BI Cloud Service Mark Rittman, CTO, Rittman Mead Oracle University Expert SummitDubai, February 2015



Deploying OBIEE in the Cloud

•Many OBIEE customers are considering deploying their BIinfrastructure in the “cloud”

• “Push” and “Pull” factors in-play ‣Perceived benefits in reducing amount of on-premise servers,software installs, upfront-license costs ‣May also be prompted by migration of rest of IT infrastructureto the cloud ‣To enable business agility - deploy the system now

-And potentially reduce dependency on IT department



Enabling Business Agility - Rapid Development in the Cloud

•Example cloud use of OBIEE - accelerating the BI development process ‣Combines agile development (“ExtremeBI”) with OBIEE deployed into Amazon AWS ‣Cloud element allows development to start now ‣Options for repatriating on-premise afterwards

•Based on Amazon AWS (EC2, S3) and Oracle BI •More details at http://www.rittmanmead.com/cloudbi/

http://www.rittmanmead.com/cloudbi/



Options for Deploying OBIEE in the Cloud

•Deploy it yourself into a public cloud (Amazon AWS, Microsoft Azure, Rackspace etc) •Buy it as a service (Oracle On-Demand, multi-tenant SaaS app) •Or … - Oracle BI Cloud Service ‣OBIEE 11g built for the cloud ‣Part of Oracle Public Cloud, comes with Oracle Cloud DB ‣Thin-client data modeller ‣Tools for managing users, roles etc



Oracle Cloud Business Intelligence

•Generally Available (GA) September 2014 •Part of the wider PaaS (platform-as-a-service)offering including database, Java, documents etc

•Aimed at a very specific set of use-cases initially ‣Adding BI and analytics to cloud-based apps ‣Departmental sandbox-type analysis



Self-Service, Consumer-Style Experience



Oracle BI Cloud Service v1 Functionality



Oracle Cloud Business Intelligence - How it Works

•Runs in Oracle’s public cloud environment (http://cloud.oracle.com) •No upfront license cost ‣Pay monthly as OpEx

•Uses new thin-client RPD modelling tool •Only reports against a single Oracle database schema (Oracle Cloud Schema-as-a-Service)

• “Standard” OBIEE - but will possibly preview newfunctionality before on-premise versions

•Patched automatically by Oracle ‣Customer able to be max one version behind?

•Comes with two instances - pre-prod and prod ‣Code promotion through BI Archive export/import

Cloud

On-PremiseAdmin accessover HTTP

Oracle Schema-as-a-serviceDatabase (single source)Oracle BI-as-a-Service

http://cloud.oracle.com



Oracle BI Cloud Services v.1 Logging into BICS, Accessing Services



Logging In and Providing Identity Domain

•Login separate to oracle.com login; provided during Cloud service provisioning process • Identity Domain is the tenant ID; all users of same service use same domain ID

http://oracle.com



Oracle Cloud : My Services Dashboard

•Oracle BICS bundle includes Oracle Database Cloud Service (Schema-as-a-Service) •Provides data store for OBIEE data, with ApEX front-end and data upload •View status of services •Click on link to launch required service ‣Database ‣Business Intelligence

Click to launch BICS Service Dashboard



Oracle Business Intelligence Cloud Service : Dashboard

•View basic metrics on system usage (users, author count over period)

•Subscription details; Company name etc ‣Click on Service Instance URL to launch OBIEE Presentation Services

Click to launch OBIEE Home Page



Oracle BI Cloud Services v.1 Accessing BI Dashboards & Analyses



BI Cloud Service Home Page and Dashboards

•Answers and Dashboards (with new Oracle Cloud look-and-feel) •But with tools to upload data, build the RPD, administer users, manage system



Creating Analyses and Dashboards

•Standard UI for creating analyses, dashboards etc (but limited to subset of features now) •Each tenant has a single RPD, all users report against that (but can add security etc) •Helpers when user first connects to guide through key features



Standard Dashboard and Analysis Editor Experience

•Regular Presentation Services interface - Dashboard, Analysis Editor, Catalog view etc •Uses new “Skyros” theme from Fusion Apps



Dashboard and Analysis Limitations in v1

•Can create analyses, dashboards, use filters, selection steps, calculations etc •Can create dashboard prompts, filters, use repository and session variables etc •No access to Agents, Scorecards, BI Publisher (anything that uses NQ_ tables) •No Essbase, SOA / EBS integration etc •No export ofcatalog objects



Oracle BI Cloud Services v.1 Uploading data to Database Cloud Service



Oracle Database Cloud Service

•Each instance of BICS comes with Oracle Database Cloud Service (Schema-as-a-Service) •ApEx environment with 10GB storage attached •Able to run PL/SQL packages (with 5m timeout) •Either create tables, views etc from ApEX,or use data uploader in BICS

Click to launch ApEX Home Page



ApEx Development Environment

•Full ApEx environment for application building, table creation, SQL queries •Can be used to build supporting applications, administration screens for OBIEE application •Make use of PL/SQL functionality ‣Data cleansing ‣Call Web Service REST APIs ‣More complex processingand calculations

•DB Cloud Service dashboard has toolsfor data and application exports



Options for Uploading Data to BI Cloud Service (DB Cloud)

•Use ApEx front-end and tools within it (data upload, data modeller etc) •Use SQL*Developer and SFTP data upload facility •Use BI Cloud Service Data Uploader



SQL*Developer for Database Cloud Access / Upload

•SQL*Developer 4+ can view, query and upload data to Database Cloud •Uses DB Cloud REST API to access database •Create “carts” of data to upload •Uses SFTP to bulk-upload data imports to cloud



Data Uploader in ApEx

•Upload spreadsheet, XML data, text (delimited or fixed width) •Store as a table in Database Cloud Service •Create keys, joins, views etc



BI Cloud Services Data Uploader

•Custom ApEx application shipped with DB Cloud Service for BICS •Upload CSV, text, xls and xlsx files to Database Cloud •Paste data from clipboard and upload •Limited data re-formatting features •Preferred method for uploading data

Click to launch Data Uploader



BICS Data Uploader Step 1 : Select File to Upload

•Press Upload to upload txt, csv, xlsx or xls file to application •Contents of file then appear inPreview window •Or just paste data from clipboardinto same preview window

•Option to ignore headers line •Option to select delimiter char



BICS Data Uploader Step 2 : Select Target or New Table

•Either select existing table to upload to, or enter details for new table • Incoming columns in file mapped to table columns •Ability to change column case, strip spaces, format as number



BICS Data Uploader Step 3 : Preview Content, and Upload

•Preview file contents with transformations and final column names •Upload file to Database Cloud Service - is the used to load a regular DB table •Review table and check for loading errors



Oracle BI Cloud Services v.1 Creating the repository



Thin-Client Data Modeller

•New thin-client data modeller, works with RPD online •Single repository per tenant •Regular RPD, but modeller only supports subset of BI Administration features •Single LTS per logical table •No Alias, have to create DB views •Single business model •Only logical calculations •But otherwise - an RPD

Click to launch Data Modeller



Data Modeller Repository Creation Workflow

1. Add tables to Database Cloud Service via data uploader 2. Lock Model (RPD) 3. Add joins if not automatically introspected 4. Select either a dimension or fact to create 5. Select table to base fact or dimension on 6. Format logical column names, values 7. Add calculations, aggregations 8. Add hierarchies 9. Publish model and unlock



Logical Fact Table Features

•Joins to dimensions automatically introspected when tables selected for model •Add default aggregation for logical columns •Define logical calculations(post-aggregation calcs only)

•Add data filters against application roles

•Set access permissions for application roles



Logical Dimension Table Features

•Alter and rename default logical column names •Add new logical columns using logical SQL expressions •Add one or more level-based hierarchies



Variables

•Can create static and dynamic repository variables •Also create session variables, including access to system session variables ‣Access to :USER etc system session variables



Oracle BI Cloud Services v.1 Administration and Maintenance



Manage Users and Application Roles

•Click on Manage link on Homepage, launches BICS Service Console •View users, assign to roles •Map roles to application roles defined by tenant administrator

Click to launch BICS Service Console



Snapshots (BI Archives)

•Entire system (RPD, catalog) can be snapshotted with versions stored online •Download and upload snapshots ‣Test to production migration ‣Restore snapshots for versioning

•Note - overwrites entire RPD, catalog ‣Does not include DB Cloud elements,need to be archived separately



Oracle BI Cloud Services v.1 Example Application : Cloud CRM Reporting



Example Application #1 : Cloud CRM Reporting - Scenario

•Many customers use SaaS applications for CRM, sales automation, talent management etc •Often silos of information, and hard to report against ‣Access via REST APIs rather than SQL / ETL

•BI Tools needed, but prefer if SaaS deployed too •Solution : Oracle BI Cloud Services,but how to access SaaS data sources?



SaaS Data Access Solution : DB Cloud PL/SQL REST Access

•Database Cloud Service supports PL/SQL packages, procedures •ApEx has APEX_WEB_SERVICE API, supports RESTful and SOAP web service calls •Salesforce.com provides REST API for retrieving objects, data ‣PL/SQL package retrievesSalesforce instance data ‣Stages into Database CloudSchema for OBIEE access

•Need to work around 5 min timeout for PL/SQL prods ‣DB Cloud limitation ‣Break retrieval into small steps, <1 min

BEGIN apex_web_service.g_request_headers(1).name := 'Accept'; apex_web_service.g_request_headers(1).value := 'application/xml'; l_clob := apex_web_service.make_rest_request( p_url => 'https://login.salesforce.com/services/oauth2/token?grant_type=password&client_id=' ||clientId||'&client_secret='||clientSecret||'&username='||username|| '&password='||password, p_http_method => 'POST' , p_body => ''); XMLData := XMLType(l_clob); FOR r IN ( SELECT ExtractValue(Value(p),'/OAuth/instance_url/text()') AS r_instance , ExtractValue(Value(p),'/OAuth/access_token/text()') AS r_token FROM TABLE(XMLSequence(Extract(XMLData,'/OAuth'))) p ) LOOP



Build Repository Against Database Tables

•PL/SQL Package calling Salesforce.com REST API populates Oracle tables •Oracle Tables then mapped into BI repository •Facts and dimensions created •Variables defined •Users added into roles



Supporting SaaS Security Models

•Full import of SaaS dataset presents the problem - how do we then respect app security? ‣Users will connect into OBIEE as themselves, no online access to SaaS security

•Solution : download user > account security objects,join those to facts to perform row-level filtering ‣Needs knowledge of underlyingSaaS row-level security setup ‣Join security access table to modelto apply data restrictions ‣But then allows BICS app to workdisconnected from SaaS application,but still apply correct security rules



Example Dashboard using Salesforce.com Dataset

New graph type: Example of visualisation appearing first on BICS





Thank You for Attending!

•Thank you for attending this presentation, and more information can be found at http://www.rittmanmead.com

•Contact us at [email protected] or [email protected] •Look out for our book, “Oracle Business Intelligence Developers Guide” out now! •Follow-us on Twitter (@rittmanmead) or Facebook (facebook.com/rittmanmead)






Oracle BI Cloud Service Mark Rittman, CTO, Rittman Mead Oracle University Expert SummitDubai, February 2015



Oracle Essbase and Oracle BI EE 11g Integration Tips and Techniques Mark Rittman, CTO, Rittman Mead Oracle University Expert SummitDubai, February 2015



OBIEE 11g: Where Are We Now?

•OBIEE 11.1.1.7 release came out in April 2013 - many new features + updated look and feel •Fourth major release of OBIEE 11g since the initial launch back in 2010 •Enterprise BI platform centered around the Common Enterprise Semantic Model (RPD) •Mobile BI apps, MS Office integration, ad-hoc, dashboard and published reporting

•Built around Oracle Fusion Middleware •Deployable on Windows, Unix, Linux •Accessing a range of enterprise data sources ‣Oracle and other RDBMSs ‣Essbase and other OLAP servers ‣Files, XML, web services ‣ADF and SOA sources ‣TimesTen in-memory database



Embedded and Leveraged Throughout the Oracle Tech and Apps Stack

•Packaged EPM and CRM analytics through Oracle Business Intelligence Applications 11.1.1.7.1

•Embedded in the Oracle Fusion Applications •Available as BI components within Oracle Fusion Middleware



Part of the Wider Oracle BI Foundation Suite 11g

•Typically now sold packaged as Oracle BI Foundation Suite 11g ‣Includes Oracle BI EE (dashboard analytics + enterprise semantic model) ‣Oracle BI Mobile ‣Oracle Scorecard & Strategy Management ‣Oracle Essbase Server

•This provides the context, and the driver, forintegration between Essbase and OBIEE ‣Essbase is the “go-to” OLAP server for Oracle BI EE, and Oracle’s analytic apps ‣It’s also the standard software bundlewhen licensing Oracle Exalytics



Runs Better on Oracle Exalytics In-Memory Machine

•Engineered System for running Oracle’s BI and EPM products •1TB RAM, 40 CPU cores, Oracle Linux 64-bit OS •Optimizations to move more of BI + OLAP processing in-memory •Lower TCO through co-ordinated patching, support,software releases

•Comes with OBIEE, Essbase (ASO + BSO),TT in-memory cache

Exalytics In-Memory Machine

Spans Relational, Multi-Dimensional, and Unstructured analysis, combined with Financial & Operational Planning ‣ In-Memory Optimized Hardware ‣ In-Memory Oracle BI, TimesTen, Essbase, and Endeca ‣ Many In-Memory Software Innovations

Tightly-Integrated with Exadata, and Big Data Appliance



The Long-Standing Opportunity (and Threat?) from OBI / Essbase Integration

•From the point that Oracle acquired Hyperion, thetalk was always around integrating EPM and OBIEE

•Seen by many as an opportunity ‣Swap-out old Brio/Hyperion web tools for OBIEE ‣Make Essbase data available to a wider audience ‣Deliver world-class reporting tools able toaccess both relational, and OLAP data source

•But also as a risk or threat ‣Make Essbase relational (e.g. Oracle OLAP) ‣Dumb-down the Hyperion tools ‣Oracle gets side-tracked into infrastructure work ‣Lose whats good about Essbase, Planning etc

•So how’s it gone, and where are we now?



High-Level Integration Goals (and for All Oracle’s BI Platform Integration)

•Make Essbase available as a data source for OBIEE, and vice-versa ‣Also applies to Endeca Information Discovery, TimesTen etc

•Make Essbase an option for OBIEE aggregate persistence ‣Speed up ad-hoc reporting, ‣Support agile BI development ‣Maximise the value in a customer’s Oracle BI deployment

•Deploy Essbase alongside OBIEE as part of the BI product install ‣Do all the product wiring, resolve security etc differences ‣Make Essbase use as seamless as possible for BI developers / users

•Preserve the benefits of standalone Oracle EPM Suite ‣Recognize that most users of Essbase use it for Planning etc ‣Support running EPM in two modes; as part of BI, or as EPM Suite



Essbase and OBIEE Data Source Integration

•The first widely-used example of OBIEE / Essbase Integration •Oracle BI EE’s BI Server can connect to Essbase and use as data source ‣Generates MDX queries, translates to/from OBIEE’s relational model ‣Ability to call-out to specific MDX functions if required; access to aliases and other Essbase oddities ‣“It’s been a journey” - initially underwheming, much better now, but still with some issues

•OBIEE as a data source for Essbase, via Studio ‣Leverage OBIEE data integration + metadata

•Support for HFM as a data source, but not Planning •Works well, but suits Essbase best when in asales analysis / star-schema context ‣But a great way to get Essbase data to a wider, dashboard-style audience



Introduced with OBIEE 11.1.1.6.2 BP1 : Aggregate Persistence in Essbase

•OBIEE’s Administration tool has the ability to define aggregates, and then persist them in an RDBMS

•Repository then uses these aggregates to speed up dashboard queries that otherwise aggregate on-the-fly

•Basis of the Summary Advisor used with •Oracle Exalytics In-Memory Machine •OBIEE 11.1.1.6.2 BP1 bundle patch enabledstorage of these aggregates in an Essbase ASO database, using headless Essbase Studio and OBIEE Admin tool Oracle BI Server

Component

Physical SQL, MDX

Physical MDX

BI Presentation ServerComponent

Oracle BI Repository Logical Business Model

Read model

Logical SQL Results

HTTP(S)

User requestsfrom dashboard, ad-hoc queries etc

Pre-computed aggregated data, stored in an Essbase ASO database

Detail-level, and dynamically-calculatedaggregate data



So How Did OBIEE 11.1.1.6.2 BP1 Aggregate Persistence in Essbase Work?

•Developer/Administrator selects slice of business model to aggregate •Aggregate Persistence Wizard then creates a logical SQL script, processed by BI Server using nqcmd

•BI Server uses Essbase Studio dmaservlet to create ASO outline, and rules file •Detail-level data sourced via BI Server logical model through to source databases

Oracle BI Repository Logical Business Model

Oracle BI ServerComponent

nqcmd scriptcontaining logicalDDL and DML forcreating and populatingaggregate tables

Logical SQLvia BI ServerODBC interface

Physical SQL, MDX

Create outline

Read model Add aggregates

Essbase Studio dmaservlet

XML request SQL queries

Data load viaEssbase rules file

Essbase Server

Developer usesBI Administratortool to select aggregates for creation, with an output of a logical SQL script for DDL and DML

1

2

3

4

5



Aggregate Persistence Benefits & Limitations

• It worked! (sort of) •OBIEE developers had the choice of Essbase ASO, or relational storage, for their aggregates •Transparent to the developer and end-user: BI Server and Essbase Studio took care of the details

•But ... it was a strange way to create an Essbase cube ‣Separate ASO database per aggregate ‣Rather than one ASO database for entire OBIEE cube ‣Also restricted Essbase (in this install type) tojust this use, hard-wired to OBIEE using a single user ‣More of a PoC than a full feature



The First Integration of Essbase into the OBIEE Tech Stack

•What was more interesting, was how Oracle integrated Essbase into the OBIEE tech stack •OBIEE 11.1.1.6, and the 11.1.1.6.2 BP1 bundle patch, included Essbase + administrationtools as part of the install bundle, but these were not by default enabled

•Needed to be patched to 11.1.1.6.2 BP1, then installed using options in a response file

[DATAMART_AUTOMATION] ESSBASE_STUDIO_URL = "http://localhost:9080/dma/dmaservlet"; ESSBASE_SERVER = "localhost"; DMA_DATABASE = "DMA_DB";

http://localhost:9080/dma/dmaservlet



New in OBIEE 11.1.1.7 : Full Integration of Essbase into OBIEE Tech Stack

•From OBIEE 11.1.1.7, Essbase is now an install option within the OBIEE product installer ‣Has to be licensed separately, or as part of BI Foundation

• Installs Essbase Server, Essbase Studio, Financial Reporting and other BI-related/complementary tools alongside OBIEE

•Management of Essbase Server, Security, start/stop etcall from single Enterprise Manager farm



Essbase and OBIEE within Oracle BI Domain - EM Management

•Essbase can be stopped, started, restarted from EM via OPMN •Essbase metrics reported on from EM • Integrated security tools via OPSS application roles and policies



Essbase Role within Oracle BI Foundation Suite

•Essbase is deployed alongside OBIEE to complement OBIEE’s existing relational capabilities •Adds an enterprise-class OLAP server to the stack, deeply integrated •Essbase is supporting BI; elsewhere, in EPM Suite, it runs the show alone •All integration work in this context is about making BI run better,not replacing Essbase-specific tools or removing EPM Suite



Integration of Essbase with Fusion Middleware (OBIEE) Security

•Essbase security in this context is aligned with FMW security ‣Application roles and policies, use of WLS LDAP server vs. Essbase Native / HSS ‣Extensible via WebLogic providers / OPSS

•Fusion Middleware Control used for almost all Essbase security tasks ‣Data and Metadata filters still defined using EAS, provisioned using EM using application policies

•Shared Services still exists for EPM Suite, but cannot be used to integrate with OBIEE in the same way as this



Creating Essbase-Related Application Roles, and Assigning Filters

•The FMW policy store, and application roles/policies, replace Shared Services for assigning filters

•Filters still created using MaxL or EAS, but now granted through application policy resource permissions

1

2



Application Roles and Data / Metadata Filtering

•Users, groups and application roles have the Essbase-related application policies assigned to them ‣Note - need to type in resource names, do not auto-populate with list of Essbase filters

•RPD database connection pools use SSO option to pass credentials to Essbase, rather than user/password

3

4

5



The Return of EPM Workspace Integration from OBIEE 11.1.1.7

•EPM Workspace integration returns with OBIEE 11.1.1.7! •SSO via shared OPSS (FMW) security •Launch BI content from within Workspace •Store FR reports in Catalog (must launch fromEPM Workspace though, not from OBIEE dashboard)



Also - Full Essbase Cube Spin-Off in OBIEE 11.1.1.7

•More of a developer preview at this stage, but ... • ... OBIEE logical SQL command CREATE CUBE providesability to create whole Essbase ASO or BSO cube from RPD

•Rather than just a single aggregate slice •One for the future - full OLAP data mart automation?



Essbase Cube Spin-Off and Aggregate Persistence / Data Mart Automation

•Cube Spin-Off process uses OBIEE aggregate persistence feature •Maps cube into RPD physical layer, and as properly-mapped business model LTSs



And for a Bonus: SmartView Compatibility across Essbase and OBIEE

•From OBIEE 11.1.1.7, SmartView can now be used with OBIEE for Office integration •Ability to create new reports as well as analyze Catalog content via Excel, Word etc •Replaces BI Office (and original SmartView from OBIEE 10g)



Creating a SmartView Connection to OBIEE Pt.1:

•OBIEE connections generally creates as “private connections” • Install Smartview (64-bit MS Office requires MOS patch 16238382) •Select SmartView > Panel > Private Connections > Create New Connection



Creating a SmartView Connection to OBIEE Pt.2:

•Enter connection URL to OBIEE’s SmartView provider in the formathttp://[machine_name:port]/analytics/jbips

1

2



Creating a SmartView Connection Pt.3:

•Enter connection credentials for OBIEE, save as a private connection

1

2



Opening an OBIEE Analysis within SmartView

•Existing Catalog analyses can be opened and viewed, similar to BI Office



Creating a New SmartView Analysis

•View Designer feature allows new reports to be created from scratch in MS Office



Saving Back to the Catalog

•Content created using SmartView can then be saved back to the catalog, tothen be viewed as a regular BI analysis

•Or saved to the local filesystem, just like any other Excel file



Summary

•Essbase integration with OBIEE has come a long way since the Hyperion acquisition •Use-cases have crystallized around Essbase providing the OLAP server for OBIEE •Has required changes (or alternatives) to how Essbase security works, to align with OBIEE •Progress also made on having Essbase work as an automated data mart engine for OBIEE •EPM Workspace is back, but really in the context of BI - not full Planning etc integration •Starting to deliver now on the promise of Essbase taking OBIEE to the next level

•Perfect complement to Exalytics - in-memory ROLAP/OLAP with low TCO



Oracle Essbase and Oracle BI EE 11g Integration Tips and Techniques Mark Rittman, CTO, Rittman Mead Oracle University Expert SummitDubai, February 2015



OBIEE 11g and Predictive Analytics, Hadoop & Big Data Mark Rittman, CTO, Rittman Mead Oracle University Expert SummitDubai, February 2015



Adding Advanced Analytics + Big Data to OBIEE11g

•Most OBIEE applications focus on historic, “what happened” datasets •Advanced analytics + big data can help us understand “why”, and predict the future behaviour, sales activity etc ‣Targeting the right customer with the right offer ‣Discovering hidden customer segments ‣Finding most profitable selling opportunities ‣Anticipating and preventing customer churn ‣Exploiting the full 360 degree customer opportunity ‣Security and suspicious activity detection ‣Understanding sentiments in customer conversations ‣Reducing medical errors & improving quality of health ‣Understanding influencers in social networks



Oracle’s Advanced Analytics Platform



Advanced Analytics and OBIEE



What is R, and Oracle R Enterprise?

•R is a statistical language similar to Base SAS, or SPSS

•Open-source, run by the R Project (http://www.r-project.org)

•R environment is a suite of client/server products for statistical data manipulation and graphical analysis

•Modeling and Analysis performed in-memory using “frames”

•Enhanced by community-contributed packages •R distribute the open-source version of R with Oracle Linux

•Oracle R Enterprise extends R to allow analysis against frames stored in Oracle tables, viewsand embed R scripts in database PL/SQL packages



Typical R Graphical Output



Capabilities of R Compared to SQL (Built-In Stats Functions)

•R provides a wide variety of statistical and graphical techniques •Linear and non-linear modeling, classical statistical tests, time-series analysis •Classification, clustering and other capabilities •Matrix arithmetic, with scalar, vector, matrices, list and data frame (aka table) structures •Extensible through community-contributed packages, and interacts with C++, Java etc •Available for Oracle Database 11gR2 through the Advanced Analytics Option •Extends the (free) SQL statistical capabilities provided by Oracle Database ‣Ranking, Windowing, Reporting ‣Lag/Lead, First/Last ‣Linear Regression, Inverse Percentile etc



Oracle R Enterprise

•Regular R is constrained by only working with in-memory datasets (frames) •Data from tables and other database structures has to be loaded into memory •Oracle R Enterprise (ORE) removes this constraint by allowing frames to reside in DB •Automatically exploits database parallelism, plus Oracle scalability / resilience •ORE provides three key areas of functionality ‣Embedded R ‣In-Database Statistics Engine(R extensions for Oracle SQL) ‣Transparency Layer(access RBDMS-based frames)



OBIEE SampleApp Flight Delays Dashboards

• “Airline On-Time Performance and Causes of Flight Delays” dataset •Provide by Bureau of Transportation Statistics, Research and Innovative Technology, United States Department of Transportation

•Dataset containing 123M rows of non-stop US domestic flight legs

•Can we use R, and ORE, to predict delays on future flights?



Typical R Predictive Analytics Flight Delays Questions

•Are some airports more prone to delays than others? •Are some days of the week likely to see fewer delays than others? •Are these differences significant? •How do arrival delay distributions differ for the best and worst 3 airlines compared to the industry?

•Are there significant differences among airlines? •For American Airlines, how has the distribution of delays for departures and arrivals evolved over time?

•How do average annual arrival delays compare across select airlines? •What is the underlying trend for each airline?



Preparing the Dataset for R, and Running R Queries

•Create R frames using datafrom Oracle RDBMS, usingORE transparency layer

•Create R queries to manipulateflight delays data

•Build regression models •Score and rank data •40 cores and 1TB RAM in Exalyticsallows multiple R engines to be spawned, processing largerdatasets than desktop workstationcould support

ontimeSubset <- subset(ONTIME_S, UNIQUECARRIER %in% c("AA", "AS", "CO", "DL","WN","NW")) res22 <- with(ontimeSubset, tapply(ARRDELAY, list(UNIQUECARRIER, YEAR), mean, na.rm = TRUE)) g_range <- range(0, res22, na.rm = TRUE) rindex <- seq_len(nrow(res22)) cindex <- seq_len(ncol(res22)) par(mfrow = c(2,3)) for(i in rindex) { temp <- data.frame(index = cindex, avg_delay = res22[i,]) plot(avg_delay ~ index, data = temp, col = "black", axes = FALSE, ylim = g_range, xlab = "", ylab = "", main = attr(res22, "dimnames")[[1]][i]) axis(1, at = cindex, labels = attr(res22, "dimnames")[[2]]) axis(2, at = 0:ceiling(g_range[2])) abline(lm(avg_delay ~ index, data = temp), col = "green") lines(lowess(temp$index, temp$avg_delay), col="red")}



Integrating with OBIEE and Oracle BI Publisher

•R scripts can be embedded in BI Publisher data models •Results returned as image vectors in XML, and rendered as BI Publisher output •R scripts can also be referenced in functions etc and included in OBIEE RPD



R Analysis Output within the OBIEE Dashboard

Display flight delay per airport for top N busiest airports with parameters that are passed to live R engines, using R script in BIP data model

Regression analysis used to predict average delay for a route, using ORE integration within OBIEE BI Repository



DemoPredictive Analytics in OBIEE SampleApp



Data Analysis is Changing…

• In the past, it’s been sufficient to just consider transactional data for analysis & reporting •Now, customers expect to consider all their data when making key business decisions



Oracle’s Big Data Products

•Oracle Big Data Appliance ‣Optimized hardware for Hadoop processing ‣Cloudera Distribution incl. Hadoop ‣Oracle Big Data Connectors, ODI etc

•Oracle Big Data Connectors •Oracle Big Data SQL •Oracle NoSQL Database •Oracle Data Integrator •Oracle R Distribution •OBIEE, BI Publisher and Endeca Info Discovery



Part of the Wider Engineered Systems Platform



Introducing Hadoop

•A new approach to data processing and data storage •Rather than a small number of large, powerful servers, it spreads processing overlarge numbers of small, cheap, redundant servers

•Spreads the data you’re processing over lots of distributed nodes

•Has scheduling/workload process that sends parts of a job to each of the nodes- a bit like Oracle Parallel Execution

•And does the processing where the data sits - a bit like Exadata storage servers

•Shared-nothing architecture •Low-cost and highly horizontal scalable

Job Tracker

Task Tracker Task Tracker Task Tracker Task Tracker

Data Node Data Node Task Tracker Task Tracker



New Oracle Information Management Ref ArchitectureActionable

Events

Event Engine Data Reservoir

Data Factory Enterprise Information Store

Reporting

Discovery Lab

Actionable Information

ActionableInsights

Input Events

Execution

Innovation

Discovery Output

Events & Data

Structured Enterprise Data

Other Data



Oracle & Hadoop Use-Cases

•Use Hadoop as a low-cost, horizontally-scalable DW archive •Use Hadoop, Hive and MapReduce for low-cost ETL staging •Support standalone-Hadoop analysis with Oracle reference data •Extend the DW with new data sources, datatypes, detail-level data



Oracle Business Analytics and Big Data Sources

•OBIEE 11g can also make use of big data sources ‣OBIEE 11.1.1.7+ supports Hive/Hadoop as a data source ‣Oracle R Enterprise can expose R models through DB functions, columns ‣Oracle Exalytics has InfiniBand connectivity to Oracle BDA

•Endeca Information Discovery can analyze unstructured and semi-structured sources ‣Increasingly tighter-integration betweenOBIEE and Endeca



OBIEE for Enterprise Analysis Across all Data Sources

•Dashboards, analyses, OLAP analytics, scorecards, published reporting, mobile

•Presented as an integrated business semantic model •Optional mid-tier query acceleration using Oracle Exalytics In-Memory Machine

•Access data from RBDMS, applications, Hadoop, OLAP, ADF BCs etc

Enterprise SemanticBusiness Model

Business PresentationLayer (Reports, Dashboards)

In-Memory Caching Layer

ApplicationSources

Hadoop /NoSQL Sources

DW / OLAP Sources



OBIEE 11.1.1.7 : Hadoop Connectivity through Hive

•MapReduce jobs are typically written in Java, but Hive can make this simpler •Hive is a query environment over Hadoop/MapReduce to support SQL-like queries •Hive server accepts HiveQL queries via HiveODBC or HiveJDBC, automaticallycreates MapReduce jobs against data previously loaded into the Hive HDFS tables

•Approach used by ODI and OBIEE to gain access to Hadoop data •Allows Hadoop data to be accessed just like any other data source



Importing Hadoop/Hive Metadata into RPD

•HiveODBC driver has to be installed into Windows environment, so that BI Administration tool can connect to Hive and return table metadata

• Import as ODBC datasource, change physical DB type to Apache Hadoop afterwards •Note that OBIEE queries cannot span >1 Hive schema (no table prefixes)

1

2

3



Set up ODBC Connection at the OBIEE Server

•OBIEE 11.1.1.7+ ships with HiveODBC drivers, need to use 7.x versions though (only Linux supported)

•Configure the ODBC connection in odbc.ini, name needs to match RPD ODBC name •BI Server should then be able to connect to the Hive server, and Hadoop/MapReduce

[ODBC Data Sources] AnalyticsWeb=Oracle BI Server Cluster=Oracle BI Server SSL_Sample=Oracle BI Server bigdatalite=Oracle 7.1 Apache Hive Wire Protocol

[bigdatalite] Driver=/u01/app/Middleware/Oracle_BI1/common/ODBC/ Merant/7.0.1/lib/ARhive27.so Description=Oracle 7.1 Apache Hive Wire ProtocolArraySize=16384 Database=default DefaultLongDataBuffLen=1024 EnableLongDataBuffLen=1024 EnableDescribeParam=0 Hostname=bigdatalite LoginTimeout=30 MaxVarcharSize=2000 PortNumber=10000 RemoveColumnQualifiers=0 StringDescribeType=12 TransactionMode=0 UseCurrentSchema=0



Oracle Big Data SQL

•Part of Oracle Big Data 4.0 (BDA-only) ‣Also requires Oracle Database 12c, Oracle Exadata Database Machine

•Extends Oracle Data Dictionary to cover Hive •Extends Oracle SQL and SmartScan to Hadoop •Extends Oracle Security Model over Hadoop ‣Fine-grained access control ‣Data redaction, data masking ‣Uses fast c-based readers where possible(vs. Hive MapReduce generation) ‣Map Hadoop parallelism to Oracle PQ ‣Big Data SQL engine works on top of YARN ‣Like Spark, Tez, MR2

Exadata Storage Servers

HadoopCluster

Exadata DatabaseServer

Oracle Big Data SQL

SQL Queries

SmartScan SmartScan



View Hive Table Metadata in the Oracle Data Dictionary

•Oracle Database 12c 12.1.0.2.0 with Big Data SQL option can view Hive table metadata ‣Linked by Exadata configuration steps to one or more BDA clusters

•DBA_HIVE_TABLES and USER_HIVE_TABLES exposes Hive metadata •Oracle SQL*Developer 4.0.3, with Cloudera Hive drivers, can connect to Hive metastore

SQL> col database_name for a30 SQL> col table_name for a30 SQL> select database_name, table_name 2 from dba_hive_tables;

DATABASE_NAME TABLE_NAME ------------------------------ ------------------------------ default access_per_post default access_per_post_categories default access_per_post_full default apachelog default categories default countries default cust default hive_raw_apache_access_log



Big Data SQL Server Dataflow

•Read data from HDFS Data Node ‣Direct-path reads ‣C-based readers when possible ‣Use native Hadoop classes otherwise

•Translate bytes to Oracle

•Apply SmartScan to Oracle bytes ‣Apply filters ‣Project columns ‣Parse JSON/XML ‣Score models Disks%

Data$Node$

Big$Data$SQL$Server$

External$Table$Services$

Smart$Scan$

RecordReader%

SerDe%

10110010%

10110010%

10110010%

1%

2%

3%

1

2

3



Hive Access through Oracle External Tables + Hive Driver

•Big Data SQL accesses Hive tables through external table mechanism ‣ORACLE_HIVE external table type imports Hive metastore metadata ‣ORACLE_HDFS requires metadata to be specified

•Access parameters cluster and tablename specify Hive table source and BDA cluster

CREATE TABLE access_per_post_categories( hostname varchar2(100), request_date varchar2(100), post_id varchar2(10), title varchar2(200), author varchar2(100), category varchar2(100), ip_integer number) organization external (type oracle_hive default directory default_dir access parameters(com.oracle.bigdata.tablename=default.access_per_post_categories));



Use Rich Oracle SQL Dialect over Hadoop (Hive) Data

•Ranking Functions ‣rank, dense_rank, cume_dist, percent_rank, ntile

•Window Aggregate Functions ‣Avg, sum, min, max, count, variance, first_value, last_value

•LAG/LEAD Functions •Reporting Aggregate Functions ‣Sum, Avg, ratio_to_report

•Statistical Aggregates ‣Correlation, linear regression family, covariance

•Linear Regression ‣Fitting of ordinary-least-squares regression line to set of number pairs

•Descriptive Statistics •Correlations ‣Pearson’s correlation coefficients

•Crosstabs ‣Chi squared, phi coefficinet

•Hypothesis Testing ‣Student t-test, Bionomal test

•Distribution ‣Anderson-Darling test - etc.



Leverages Hive Metastore for Hadoop Java Access Classes

•As with other next-gen SQL access layers, uses common Hive metastore table metadata •Provides route to underlying Hadoop data for Oracle Big Data SQL c-based SmartScan



Extending SmartScan, and Oracle SQL, Across All Data

•Brings query-offloading features of Exadatato Oracle Big Data Appliance

•Query across both Oracle and Hadoop sources • Intelligent query optimisation applies SmartScanclose to ALL data

•Use same SQL dialect across both sources •Apply same security rules, policies, user access rights across both sources



Example : Using Big Data SQL to Add Dimensions to Hive Data

•We want to add country and post details to a Hive table containing page accesses •Post and Country details are stored in Oracle RBDMS reference tables

Hive Weblog Activity tableOracle Dimension lookup tables

Combined output in report form



Create ORACLE_HIVE External Table over Hive Table

•Use the ORACLE_HIVE access driver type to create Oracle external table over Hive table •ACCESS_PER_POST_EXTTAB now appears in Oracle data dictionary



Import Oracle Tables, Create RPD joining Tables Together

•No need to use Hive ODBC drivers - Oracle OCI connection instead •No issue around HiveServer1 vs HiveServer2; also Big Data SQL handles authenticationwith Hadoop cluster in background, Kerberos etc

•Transparent to OBIEE - all appear as Oracle tables •Join across schemas if required



Create Physical Data Model from Imported Table Metadata

•Join ORACLE_HIVE external table containing log data, to reference tables from Oracle DB



Create Business Model and Presentation Layers

•Map incoming physical tables into a star schema •Add aggregation method for fact measures •Add logical keys for logical dimension tables •Remove columns from fact table that aren’t measures



Create Initial Analyses Against Combined Dataset

•Create analyses usingfull SQL features

•Access to Oracle RDBMSAdvanced Analytics functionsthrough EVALUATE,EVALUATE_AGGR etc

•Big Data SQL SmartScan featureprovides fast, ad-hoc accessto Hive data, avoiding MapReduce



Oracle / Hive Query Federation at the RDBMS Level

•Oracle Big Data SQL feature (not BI Server) takes care of query federation •SQL required for fact table (web log activity) access sent to Big Data SQL agent on BDA •Only columns (projection) and rows (filtering) required to answer query sent back to Exadata •Storage Indexes used on both Exadata Storage Servers and BDA nodes to skip block reads for irrelevant data

•HDFS caching used to speed-upaccess to commonly-usedHDFS data



Access to Full Set of Oracle Join Types

•No longer restricted to HiveQL equi-joins - Big Data SQL supports all Oracle join operators •Use to join Hive data (using View over external table) to a IP range country lookup tableusing BETWEEN join operator



Add In Time Dimension Table

•Enables time-series reporting; pre-req for forecasting (linear regression-type queries) •Map to Date field in view over ORACLE_HIVE table ‣Convert incoming Hive STRING field to Oracle DATE for better time-series manipulation



Now Enables Time-Series Reporting Incl. Country Lookups



OBIEE 11g and Predictive Analytics, Hadoop & Big Data Mark Rittman, CTO, Rittman Mead Oracle University Expert SummitDubai, February 2015

Date post:	14-Jul-2015
Category:	Data & Analytics
Upload:	mark-rittman
View:	832 times
Download:	3 times

OBIEE11g Seminar by Mark Rittman for OU Expert Summit, Dubai 2015

Data & Analytics