Framework Manager Cognos 8 BI Objectives In this module we will examine: Basic Data Model Issues Modeling Enhancements Determinants Dimensional Modeling enhancements New General Features OLAP Data Stores Update to Best Practices New Features in Framework Manager Query Subject Enhancements Determinants Query Sets Tab Style Dialogs Dimensional modeling enhancements Regular Dimensions Measure dimensions (facts) Scope relationships Dimension Map Metadata Import System Tables option Performance Quality of Service Navigation and maintenance Model Reporting Property Based Search Search object references UI Changes Menu reorganization In Context right click Query Studio Report Studio Analysis Studio Cognos Viewer Event Studio Framework Manager Publish Run time model for end user consumption via the studios Data Access Relational Cubes SQLMDX More ODBCXML Files Composite Metric Studio Metric Database Framework Manger Basic Data Model Issues Metadata for Business Intelligence The main goals of modeling in Framework Manager are: An intuitive end user experience Leverage existing metadata and add value Predictable queries and results To fully leverage advanced features you should model your business view dimensionally. It is not required that you model dimensionally but it is important to understand how your model impacts the predictability of your queries and results. Do you know the answers to these questions? What is the difference between dimensional and normalized models? What is the impact of reporting of each modeling method? What is a conformed dimension? Review of Normalized vs. Dimensional Remember, operational and reporting databases have different goals and different viewpoints Operational databases (typically normalized) Fast updates (writes) Reporting can be slow Reporting data structures (typically dimensional) Fast reporting (reads) Reporting is the goal Understand what issues you can and cant address in FM Basic Rules with Normalized Design Design technique to minimize redundancy Focus on writes/updates rather than reads The needed data may be in separate databases, which cannot be joined Operational structures may be too complex for general reporting (especially in ERP systems) Often result in monolithic designs with multiple joins A single table can have multiple cardinality relationships One to many Many to many Many to one Opens the door to blind spots in join paths Double counting can easily take place There may be no long-term view of the data Reports can often take too long to run Framework Manager Over Normalized Design Where Framework Manager can help: It can hide the complexity of the underlying design It can make some very complex reports simple (multi-grained, multi-fact queries) How it typically WONT help Performance hit on the Operational Application may still be great Query cost is unchanged (i.e. more joins = longer running time) If historical data is not captured in the operational application, it still wont be available for reporting If you want to address these issues, you should start thinking about data warehousing Basic Rules of Dimensional Design Dimension tables should relate to Fact tables Dimensions should not directly relate to other dimensions Facts should not directly relate to other facts Fact attributes should only hold one type of fact Dont have a single Amount column that holds both Sales and Budget figures Dont mix different grains of fact in a single fact table Dont hold monthly sales budgets in the same table with daily sales You could hold monthly sales budgets with the aggregated sales for the month (both facts are at the month grain) There are occasionally good reasons for breaking these rules but they are rare. Advantages of Dimensional Modeling Subject Area Marts (Star Schema Groupings) Data for a specific area of the business Simpler for the end-user to understand fewer tables Collection of like fact tables connected to common dimensions Conformed dimensions prevent Islands of Information Adaptable and extensible Examine Conformed Dimensions A conformed dimension has relationships to more than one fact table. Product Dimension Time Dimension Sales Fact n Inventory Levels Fact 1..n 1..1 Manage Star Schemas in Framework Manager Product Dimension Time Dimension Sales Fact n Inventory Levels Fact 1..n 1..1 Product Dimension Time Dimension Sales Fact Namespace Inventory Levels Fact Namespace Examine Multi-Fact Queries Product Dimension Time Dimension Sales Fact n Inventory Levels Fact 1..n 1..1 Examine Stitch Queries Stitch queries send two separate queries to the data source and then merge them locally. The SQL generated contains syntax such as: coalesce full outer join Database Cognos 8 Query 1 Query 2 Report Output D1D2F1 Stitch Queries Together F2 A B C Y1 Y2 Y $18 $22 $45 Degenerate Dimension Kimball: A degenerate dimension (DD) acts as a dimension key in the fact table, however does not join to a corresponding dimension table because all its interesting attributes have already been placed in other analytic dimensions Facts Degenerate Dimension Dimension key that has no attributes In the warehouse having its own table would mean having as many rows as the fact Create a Dimension Between Two Joined Facts Implement a degenerate dimension solution. Facts Joins Inner Join (also known as equi- join) Consider a business that employs both managers and engineers - and some employees that are both. An inner join is like a union of this set; it selects the set of people that are both managers and engineers and provides information about them in both roles. Left Outer Join Consider again the business that employs both managers and engineers. A left outer join selects all of the managers, providing the information about them, but in the case of managers who are also engineers, it provides additional information about them. Joins Right Outer Join Consider again the business that employs both managers and engineers. A right outer join selects the set of all engineers, providing information about them, but in the case of engineers and also managers, it provides additional information about them. Full Outer Join A FULL OUTER join would return a record set that contains all the records from these three different sets 1, 2 and 3. For each record returned that does not have matching records in the corresponding table, a NULL value will be returned for each column selected from the corresponding table. CustomersOrders Set 2 Set 1 Set 3 Records with matching values Non-matching records Non-matching records Terminology Framework Manager Modeling Enhancements Modeling Enhancements in C8 Determinants Dimensional Modeling enhancements Regular Dimensions Measure dimensions (facts) Scope relationships Framework Manager Introducing Determinants A determinant can define either: the set of database columns (query items) that uniquely identify a set of data a set of columns that identify a non-unique set within the data Dimension Information in CRN 1.x was a means of controlling granularity but wasnt completely part of the OLAP world In Cognos 8, determinants are most closely related to the concept of keys and indexes in the database There is no concept of hierarchy in determinants, although the order in which they are specified does govern the order in which they are evaluated What is a determinant? What is a determinant? Example 1 Example 1 Database columns that uniquely identify a set of data: Year KeyMonth KeyMonth NameDay KeyDay Name Jan Sunday, Jan 1, Jan Monday, Jan 2, 2005 Non-Unique Determinants Month Key is all that is required to identify month but to query with month needs a select distinct and Month Key in the Group By Unique Determinant Day Key is the unique key of the table and can be associated to all columns in the table NameKeyAttributesUniquely IdentifiedGroup By YearYear KeyNoneNoYes MonthMonth KeyMonth NameNoYes DayDay KeyDay Name Month Key Month Name Year Key YesNo Determinants Example 2 Identifies a set of columns that identify a non-unique set within the data: Year KeyMonth KeyMonth NameDay KeyDay Name January Sunday, Jan 1, January Monday, Jan 2, 2005 What is a determinant? Example 2 Non-Unique Determinants Month Key is not enough to identify month. Resulting query needs a select distinct and the Group By containing Year Key and Month Key Unique Determinant Day Key is the unique key of the table and can be associated to all columns in the table NameKeyAttributesUniquely IdentifiedGroup By YearYear KeyNoneNoYes MonthYear Key, Month KeyMonth NameNoYes DayDay KeyDay Name Month Key Month Name Year Key YesNo Determinants When are determinants required? Determinants are absolutely required within your model when you are: Using model query subjects where there are joins at multiple levels of granularity (like time), especially facts at multiple levels Eliminates double-counting / double-summing of data Using blobs in your data The application of determinants is similar the application of Dimension Information (hierarchies) in CRN 1.x with a few changes Demo 1 Refine Determinants for Query Subject Demo : Refine Determinants for Query Subject Set up Determinants for the Time Dimension to ensure the correct rollup of data for the Product Forecast Fact and the Sales Fact. Determinant Concepts Group By Includes a select distinct in the query and the group by will contain the key for this determinant Uniquely Identified The unique key of the table. All the columns in the table are associated to this key. There is no hierarchy but determinants are evaluated in order of highest to lowest until it finds the first level the attribute exists at You help CRN by providing extra information in FM that tells what columns (attributes) are associated with what level of detail (granularity). Determinants with Multi-Grain Queries Usage: Determinants allow us to specify the correct level of rollup on a query subject when reporting with facts stored at different grains Example: Quantity fact = reported (and joined) at the day level Expected Volume fact = reported (and joined) at the month level (month is repeated so you would see a select distinct in the query and month key in the group by) Determinants on the Time Dimension allow data reported from both facts to be rolled up to their correct levels Determinants Additional Information On Import, FM looks for Primary Keys and Indexes CRN 1.1 keys and indexes only accessible in the model.xml file Cognos 8 keys and indexes manageable via determinants tab On import, Primary Key and Unique Index are flagged as Uniquely Identified Multi-part keys are recognized and treated as such on import Working with Determinants Workshop 1 Framework Manager Introducing Dimensional Objects Dimensional vs. Relational Modeling When to dimensionally model a relational source To enable relational reporting within analysis studio To enable drill up/down functionality in reports and ad hoc queries To access member functions in the report authoring tools Refer to Modeling recommendations in the Framework Manager User Guide When to use only relational modeling on a relational source: To enable basic ad hoc querying and relational reporting A purely relational model built using the best practices from CRN 1.1 will be suitable to extend using dimensional features, this can be done at any time. Refer to Modeling Best Practices from CRN 1.1 Introducing Regular Dimension Objects Regular dimension object: Contains one or more hierarchies each with several levels Additional attributes can be specified at the appropriate levels Rolls up the data between the levels Allows drill up / down functionality in the studios No ambiguity with multiple hierarchies however there is a restriction of using items from both hierarchies within the same query Requirements: Each level must contain a caption and a key Captions must be a character field Key is the query item that uniquely identifies the members in that level Demo 2 Create a Regular Dimension Demo - Create a Regular Dimension Dimension: Product Hierarchy: Product Levels: Product Line: Key(s): Product_Line_Code; Role=_businessKey Attribute(s): Product_Line; Role=_memberCaption Product Type: Key(s): Product_Type_Code; Role=_businessKey Attribute(s): Product_Type; Role=_memberCaption Product: Key(s): Product_key; Role=_businessKey Attribute(s): Product_Name; Role=_memberCaption Attribute(s): Product_Description; Role=_memberDescription Introducing Measure Dimension Objects A measure dimension is a logical collection of facts Can be created from: A single table in the database Multiple db queries combined in a model query subject Measure dimensions link relevant regular dimensions and are required to: Enable OLAP style analytical querying of Relational Data Sources Enable drill up/down functionality in reports Access to member functions in the studios Cognos 8 Facts Query Subject A query subject can be a dimension or a fact depending on its cardinality relative to other query subjects involved in the query. There is no way for an end user to distinguish between a fact and a dimension. Best Practice recommends hiding the foreign keys from users Measure Dimension Measure Dimensions are always treated as Facts Only measures are visible Measure Dimensions are joined to Regular Dimensions through scope relationships Both Measure and Regular Dimensions have very clear roles in the model. Less chance for confusion Allows for OLAP style querying Demo 3 Create a Measure Dimension Cognos 8 Relationships Query Subject Relationships Defines the physical relationship or how the underlying data is related. A relationship is a SQL concept that defines the physical JOIN between query subjects. Equivalent to the WHERE clause generated when multiple query subjects are combined in a report. Physical relationship between query subject will always be required Cardinality is still used to determine if a query subject is a fact or a dimension in context to the query Cognos 8 Scope Relationships Defines the reporting relationship between regular and measure dimensions Scope relationships exist ONLY between regular and measure dimensions. Governs the reporting granularity of the regular dimension for a particular measure (each measure in a dimension can have different scope assigned) does not impact the WHERE clause. Similar to setting scope in Transformer Scope Relationships A scope relationship defines: Defines the reporting relationship between regular and measure dimensions The inclusion/exclusion of a dimension in a star schema A scope relationship will be automatically created between a dimension and a measure dimension whose underlying query subjects have a valid JOIN relationship defined Scope can be modified or deleted and is similar to setting scope in Transformer Dimension Map View View, create, or modify hierarchies or levels; View, create or modify regular or measure dimensions; View, create or modify scope relationships for measure dimensions or individual measures Demo 4 Dimension Map UI Merging or Converting into Dimensions Merge Query Subject with Determinants results in: Original Query Subject is maintained New Regular Dimension is created using the levels Convert Query Subjects into a Regular Dimension results in: Only a Regular Dimension Object Demo 5 Merging and Converting Demo - Merging and Converting Create the Time Dimension with determinants; Merge into a Regular Dimension; Convert original Query Subject into a Regular Dimension Dimensional Modeling Workshop 2 Why Both Determinants and Regular Dimensions? Determinants For traditional query subjects Required to manage granularity for multi-fact/multi-grain reporting Required for blobs Represents levels of grouping data Does not provide OLAP functionality Cannot be used in Analysis Studio Regular Dimensions Allows you to perform OLAP analysis against Relational data (dimensionally modeled relational) Provides OLAP functionality such as drill up / down Is required for Analysis Studio Dimensional Modeling Considerations To better understand the impact of dimensional modeling we recommend you consider the following: Relational query generation continues to be defined by query subjects and joins as it was in CRN 1.1 Dimensional objects are an additional layer of metadata that enable OLAP behaviors. There is a cost to the additional capability enabled by OLAP behavior on a relational source It will most likely be necessary to employ a form of RDBMS vendor materialization to improve performance, particularly for drill through. It is highly recommended that you build mandatory filters into your model to ensure that end users do not accidentally retrieve excessively large data sets. Similar rules regarding data volumes apply to a dimensionally modeled relational source as apply to building cube. The key difference is that with filtering strategies it is possible to perform analysis against larger volumes in a relational source than is practical to do with most OLAP sources. Framework Manager OLAP Data Stores OLAP Data Store Support Setup OLAP data store in Framework Manager Setup Data source connection Import OLAP data store (essentially a link) Publish OLAP Package to Cognos Connection Administration in OLAP Administration Tool Changes to OLAP data stores are automatically consumed by Studios Supported OLAP data stores Cognos PowerCube, Microsoft Analysis Services, Hyperion Essbase or IBM DB2/OLAP SAP BW is an OLAP store An RDBMS is not an OLAP data store, even when dimensionally modeled. What about PowerPlay 7 Components? PowerPlay Transformer Series 7 Version 2 and higher continues to be used as the modeling tool for PowerPlay Cubes OLAP data stores cannot be edited within Framework Manager PowerPlay Client Series 7 is used for disconnected users who cannot access Analysis Studio PowerPlay Enterprise Server is not used with Cognos 8 as it is not necessary. Instead of connecting to cubes through PowerPlay Enterprise Server you now connect to cubes through FM published packages using MDX. PowerPlay and Cognos 8 Transformer Series 7 Version 3, MR2 has been re-branded and will be shipped with Cognos 8 New Name when shipped with C8: Cognos 8 Business Intelligence OLAP Modeling PowerPlay for Windows Series 7, Version 3 has been re-branded and will be shipped with Cognos 8 New Name when shipped with C8: Cognos 8 Business Intelligence Mobile Analysis Modeling with OLAP Data Sources Workshop 3 OLAP and Relational What you can and cant do: You can combine OLAP data stores and relational data stores in the same model and package You can create a report using both based on the following rules: JoinsSet Ops (unions, etc) Master- Detail Drill-Through OLAP to OLAP NoYes OLAP to RDBMS NoYes RDBMS to RDBMSYes Report bursting is limited: The burst report must be a grouped report and the burst is restricted to the outer most grouping in the report. i.e. report grouped on Country and State but can only be burst on Country Modeling with OLAP and Relational Data Sources Workshop 4 Framework Manager New General Features Changes Overview Performance: Quality of Service Query Sets Test Capabilities Model Reporting Diagram enhancements Property Based Search Search object references Query item properties True/False Boolean Navigate to Quality of Service Highlight a Data Source From the properties menu expand Type the Function Set ID property In Project Function List pane, click Define Quality of Service Performance - Quality of Service Define Query Set Similar to tabular sets in CRN 1.x Report Studio Requires two identical query subjects Supports union, except, intersect Query Set Union, Except, and Intersect Union Product 1 Product 2 Except Product 1 Product 2 Product 1 Product 2 Intersect Test Cognos 8 update Enhancements allow the modeler to test query scenarios without going to the studios. Batch runs the entire query - similar to running in pdf mode from studio. Test for performance and to ensure entire query runs properly from FM (as opposed to running html in studio) Auto Sum test the SQL and results with automatic aggregation on determinants Multiple Query Subject Testing select one for more query subject or query items from one or more query subject and Test is now available. Replaces the need to create model query subjects or to go into a studio to verify SQL generation. Demo 6 UI Overview Demo UI Overview Quality of Service Query Sets Testing Model Queries Model Report Property Based Search Search object references Query Item properties Logging View Transaction History Script Player Run Scripts Workshop - Query Sets and Test Enhancements Query Sets Import Views from go sales data warehouse Select the AR_AMX and AR_CEU Views to import Compare data columns between the imported views Highlight the two views From the Actions menu Define Query Set What would have happened if we selected Merge into a New Query Subject ? Test enhancements TEST: Select one or more linked Query Subjects TEST: Select query items from one or more query subjects Observe the data and the SQL Compare the Batch and AutoSum Check boxes results Framework Manager Best Practices Update Best Practices Review 1.Take a multiple layer approach to modeling 2.Customize Metadata at run-time 3.Setup Query Item properties for Business Intelligence use 4.Make sure all the relationships you need exist in the database view. Ensure there are no discrepancies in cardinality between database and modeled environment 5.Begin creating simplified, abstracted model query subjects for your business view 6.Resolve valid multiple relationships (ambiguous join paths) by generating Role Playing Dimensions 7.Use dimension information to enable automatic aggregation capabilities 8.Handle multiple hierarchies 9.Create a Business View with the Star Schema Grouping Wizard 10.Secure the Model for deployment Updates to Best Practices 1.Take a multiple layer approach to modeling 2.Customize Metadata at run-time 3.Setup Query Item properties for Business Intelligence use 4.Make sure all the relationships you need exist in the database view. Ensure that any discrepancies in cardinality or relationships between database and modeled environment are correctly reflected by determinants. 5.Begin creating simplified, abstracted model query subjects/dimensions and measures for your business view 6.Resolve valid multiple relationships (ambiguous join paths) by generating creating multiple query subjects for Role Playing Dimensions. 7.Create dimensions with levels, hierarchies and keys to enable automatic aggregation capabilities and OLAP query functionality on relational sources. Create multiple hierarchies within the dimension if desired. 8.Create measure dimensions and adjust/set scope relationships. 9.Create a Business View with the Star Schema Grouping Wizard 10.Secure the Model for deployment Summary In this module we examined: Basic Data Model Issues Modeling Enhancements Determinants Dimensional Modeling enhancements New General Features OLAP Data Stores Update to Best Practices Questions? Operational vs. Reporting Databases Two major designs for databases Operational Day to day operations Usually Normalized or a vendor (ERP) package Reporting Copy of operational data Restructured to make reporting easier and faster Usually Dimensional Operational Data Design Normalized Design (often called 3 rd Normal Form) Persistent storage of atomic data (no summaries) Maximize accuracy Minimize redundancy Maximize the speed of writes to the database ERP / Packaged Application Design Same as above, plus Maximum install and configuration flexibility, usually through the use of many recursive structure tables relationship rules Entity Relationship Model Typical normalized design Customer Type Product Type Product Customer Sales Area Sales Rep Order Header Order Line Product Line table Entity Relationship Model Data integrity legs Maximize accuracy and speed of writes Customer Type Product Type Product Customer Sales Area Sales Rep Order Header Order Line Product Line Entity Relationship Model Data integrity legs Maximize accuracy and speed of writes write check Customer Type Product Type Product Customer Sales Area Sales Rep Order Header Order Line Product Line Customer Type Product Type Product Customer Sales Area Sales Rep Order Header Order Line Product Line Entity Relationship Model Data integrity legs Slow down reads! Show all customer types that bought from a product line. Reporting Problems with a Normalized Design Reporting from normalized databases: Impact of reporting on performance may be too great Operational structures may be too complex for general reporting (especially in ERP systems) Reports can often take too long to run The needed data may be in separate databases, which cannot be joined There may be no long-term view of the data Framework Manager Over a Normalized Design Where Framework Manager can help: It can hide the complexity of the underlying design How it typically WONT help Performance hit on the Operational Application may still be great Reports may take just as long (or longer) to run Still may be no long-term view of the data New issues It can be VERY hard to simplify the complexity correctly You must know what to look for and watch out for traps Reporting Database Design Reporting database goals are much simpler than Operational database: Faster and easier access to corporate data for accurate reporting and analysis. Most common reporting design is a Dimensional Model Dimensional Database Design Dimensions Contain Descriptive information May have multiple key segments for multiple levels in one or more hierarchies The key to connecting star schema groupings Conformed dimensions enable multi-fact querying Facts Contain metrics often aggregated The center of the star schema A Dimensional Database Star Schema ProductsCustomers Sales Reps Order Fact Dates Central Fact Table Surrounding Dimension Tables Designing a Reporting Database Star Schema ProductsCustomers Sales Reps Order Fact Dates Show all customer types that bought from a product line. Creating Dimensions - Collapsing the Legs Creating Star Schemas You can collapse data integrity legs ProductsCustomers Sales Reps Order Fact Dates Customer Type Product Type Product Customer Sales Area Sales Rep Order Header Order Line Product Line Collapsing the Legs Creating Star Schemas You can collapse data integrity legs ProductsCustomers Sales Reps Order Fact Dates Customer Type Product Type Product Customer Sales Area Sales Rep Order Header Order Line Product Line Collapsing a Leg Each entry at each level is a member E.g. 2 Product Line Members, 12 Product Members 2 rows4 rows12 rows Product LineProduct TypeProducts Combining the Items You will have as many rows as the lowest entity in the Data Integrity Leg 12 rows Products Dimension Producing the Dimension There are two ways you will generally get the dimension Someone actually built a Dimension Table in a Warehouse You have a single table containing all those rows Someone constructed a Query to yield the same result set You still have the multiple tables in the database (normalized) To the user, it looks like a single table Single Table (Warehouse) Dimension Pros Address most of the issues in reporting from normalized sources Complexity, performance hit, run time, history, etc. Very easy to set up in FM for dimensional analysis Cons Often it doesnt exist; no one has built a warehouse yet No obvious hierarchy (if any) No obvious ownership (determinant) of attributes Double counting of higher level members Dimension Table Issues Is it Product Line > Product Type or Product Type > Product Line? Is Desc_1 for Product Line or Product Type? How many Product Lines? 2 or 12? Multi-Table (Query) Dimension Pros It may be the only option (no physical Dimension Table) It reduces the visible complexity Hierarchy is visible in an ERM (not necessarily to the user) Attribute ownership is obvious in an ERM. It is determined by the key of the individual tables (assuming tables are normalized) Less likely to double count; there are only 2 Product Line rows (ReportNet/UDA more likely to figure it out) Cons Most normalized reporting problems still exist performance hit, run time, history, etc. Requires significant work to set up for dimensional analysis CRN 1.x VS C8 Dimensions and Facts CRN 1.x A query subject could be a dimension or a fact depending on its cardinality relative to other query subjects. There was no way for an end user to distinguish between a fact and a dimension. Dimension Information allowed control over order of grouping but didnt allow drill up/down. Best Practice required multiple query subjects to support multiple hierarchies (reduce ambiguity) Dimension Objects Cardinality has not changed Measure and Regular dimensions joined using scope relationships Regular and Measure dimensions are different objects to Query Subjects (distinguishable OLAP objects) Dimension objects enable true dimensional OLAP style querying (drill up / down) Regular dimension objects can contain Multiple hierarchies (no ambiguity) Facts The Theory Facts are what gives dimension tables a purpose E.g. Products sold this month E.g. Customers who bought Product X Both questions require a Sales fact table Collection of measurements Foreign keys relate to dimensions Add value with calculations based on existing measures/business rules Determinants in C8 Help eliminate double counting and summing. FM 1.1 did this with keys and attributes in the Dimension Information Determinants in FM 8 do the same as Keys and Attributes in FM 1.x with some additional control True hierarchies are now available in FM8 Determinants What is a determinant? Keys /column(s) that uniquely identify a set of data. If you know the value of a determinant, you will get only one possible value for each of its determined attributes For example Employee_Number determines Last_Name. There may be many Smiths with different Employee Numbers, but Employee Number 123 will return only Smith, not Smith & Jones How do I use a determinant? The application of determinants is very similar the application of Dimension Information (hierarchies) in CRN 1.x Compare Dimension Information to Determinants Determinants in C8 On Import, FM looks for Primary Keys and Indexes PK and Unique Index are flagged as Uniquely Identified All attributes are determined by Unique Keys on Dimension Tables but additional assignment is recommended Grouping on Determinants - Example Product Fact has measures at the month level Sales fact has measures at the day level We want a group by at the month level so that when we report with measures from both facts the month level data is treated as a min (or is grouped). Why dont we do group by at the Quarter, Year levels? If we added another fact table which has measures reported at the quarter level then we must group by Quarter as well. Even if you dont have Quarter facts, you may still mark as group by for future use Determinants Override Index Assumptions In FM 1.x we imported Index information and used that information to determine grouping. The only way to override index grouping assumption was to modify the XML Determinants expose our grouping assumptions and allow us to override them. Demo: Determinants on Import All Attributes under the PK Month and Weekday attributes not assigned to any key Unique level at the top Bogus Determinants without keys or attributes Reporting Multi-Grained Facts ReportNet (both in Query Studio and Report Studio) is specifically designed to make accurate assumptions about aggregating facts (sum, average, etc.) This is true for both CRN 1.x and Cognos 8 Facts in different tables are often at different levels of granularity. For example Sales Quantities are at the Day time grain Expected Volumes are at the Month time grain Reporting Multi-Grained Facts To automatically report both sets of facts on the same report, ReportNet must understand the granularity of the facts so that it can group and summarize the higher grained fact. A report that doesnt understand multiple grains will double sum and double count. For example, the Expected Volume would appear on every row with the individual Sales Quantities instead of just once for the month. This isnt a CRN problem, this is basic data relationships and SQL reporting. Multi-Grained Reporting - Example Create Report on: Quantity from Sales Fact Expected Volume from Product Fact Dates from Time Dimension Problem: Quantity fact grain = day level Expected Volume fact grain = month level Querying two facts at different levels of granularity produces: Modeling Data in FM for Multi-Grained Facts ReportNet understands and knows how to avoid double counting with dimensional models. (This is what those, much discussed, Stitch Queries are all about.) It is therefore recommended that you model the Query Subjects in Framework Manager to look dimensional, even if it they are not (i.e. make normalized data look dimensional) For example, you should separate multi-grained fact table with a degenerate dimension Query Subject As part of that process, you help CRN by providing extra information in FM that tells what columns (attributes) are associated with what level of detail (granularity) In 1.x we did this with Dimension Information. In C8, we use determinants. Review of Multi-Grained Fact Report - Example Create Report on: Quantity from Sales Fact Expected Volume from Product Fact Dates from Time Dimension Problem: Quantity fact grain = day level Expected Volume grain = month level Querying two facts at different levels of granularity produces: Solution: Add determinants to time dimension to resolve granularity Demo: After Refinement Attributes grouped with actual determinants Missing attributes added You may rename determinants if you wish Bogus Determinants removed Unique level at the bottom select TIME_DIMENSION.MONTH_KEY as MONTH_KEY, PRODUCT_FORECAST_FACT.PRODUCT_KEY as PRODUCT_KEY, XSUM(PRODUCT_FORECAST_FACT.EXPECTED_VOLUME for TIME_DIMENSION.MONTH_KEY,PRODUCT_FORECAST_FACT.PRODUCT_KEY ) as EXPECTED_VOLUME from go_datawarehouse..dbo.TIME_DIMENSION TIME_DIMENSION join go_datawarehouse..dbo.PRODUCT_FORECAST_FACT PRODUCT_FORECAST_FACT on (TIME_DIMENSION.MONTH_KEY = PRODUCT_FORECAST_FACT.MONTH_KEY) where (TIME_DIMENSION.MONTH_KEY in (200401)) and (PRODUCT_FORECAST_FACT.PRODUCT_KEY in (30001)) group by TIME_DIMENSION.MONTH_KEY, PRODUCT_FORECAST_FACT.PRODUCT_KEY SQL with Determinants Not Set with TIME_DIMENSION3 as (select TIME_DIMENSION.MONTH_KEY as MONTH_KEY from go_datawarehouse..dbo.TIME_DIMENSION TIME_DIMENSION group by TIME_DIMENSION.MONTH_KEY ) select TIME_DIMENSION3.MONTH_KEY as MONTH_KEY, PRODUCT_FORECAST_FACT.PRODUCT_KEY as PRODUCT_KEY, XSUM(PRODUCT_FORECAST_FACT.EXPECTED_VOLUME for TIME_DIMENSION3.MONTH_KEY,PRODUCT_FORECAST_FACT.PRODUCT_KEY ) as EXPECTED_VOLUME from TIME_DIMENSION3 join go_datawarehouse..dbo.PRODUCT_FORECAST_FACT PRODUCT_FORECAST_FACT on (TIME_DIMENSION3.MONTH_KEY = PRODUCT_FORECAST_FACT.MONTH_KEY) where (TIME_DIMENSION3.MONTH_KEY in (200401)) and (PRODUCT_FORECAST_FACT.PRODUCT_KEY in (30001)) group by TIME_DIMENSION3.MONTH_KEY, PRODUCT_FORECAST_FACT.PRODUCT_KEY SQL with Determinants Set Workshop 1 (Task 1 6) Import and prepare sales metadata using query subjects Relationships Query items Multi-lingual Set up Query Items for BI Search is on the Tools pane lower right hand corner Add missing relationships Refine at least one set of determinants Create some role playing dimensions (You may use scripts, if you wish) Determinants vs. Hierarchies Determinants and Hierarchies are very similar, but there are slight technical differences Determinants Keys that determine the values of other attributes. Determinants group attributes but do not imply hierarchies Contain one view of the data granularity Hierarchies Establish relationships between groups of determined attributes Used primarily for aggregating metrics to pre-determined levels Can contain multiple hierarchies May or may not be natural hierarchies Imposing Hierarchies Over Groupings Hierarchies are imposed over groupings to aid in understanding, primarily through rollups (aggregation). A single dimension with multiple groupings may have no hierarchy, one, or more. These choices often only show up in OLAP analysis. Do Lines roll up to Brands or do Brands roll up to Lines? It depends if you are a Line Manger or a Brand Manager ProductNameCategoryBrandLine 1Pro-sportAthleticNekeShoe 2Easy WalkerRecreationalAdetusShoe 3Polo-oneRecreationalNekeShirt Category Brand Line Category Line Brand Imposing Hierarchies Over Groupings A normalized model of the data might look like the diagram below. The operational system may show this or some other model. By collapsing all the attributes into a single dimension table or FM Regular Dimension, the users may each choose their own hierarchy for analysis. Dimensional Information in FM 1.1 Hierarchy Information Really used for grouping couldnt drill up/down Not visible to report writer (unless grouped in folders) Determinant Information Specified only with dimensional information, whether user wants dimensional reporting or not C8 Regular Dimension (Hierarchy) Hierarchies dont exist in Query Subjects, only in Regular Dimensions When created by merging related query subjects, initial Levels are created from the query subject determinants Allow true drill up/down reporting (OLAP) Provide additional member roles (e.g. Caption) Why Both Determinants and Regular Dims? In 1.x, you could report details or summaries, but not perform dimensional analysis. You reported from databases and did OLAP (with PowerPlay) on cubes In Cognos 8 we have uncoupled our reporting methods (detailed reporting or OLAP) from data stores (databases or cubes) RS and QS provide detailed reporting from both databases and cubes Regular Dimensions allow OLAP against both cubes and databases (yes, ROLAP, just not aggregate aware) Why Both Determinants and Regular Dims? Regular Dimensions represent true awareness of levels and hierarchies Enable OLAP style analytical querying of Relational Data Sources (ROLAP) Enable drill up/down functionality in reports Visible representation of multiple hierarchies Improved control over hierarchy rollups Access to member functions in the studios Model vs. Data Source Regular Dimensions Regular Dimension hierarchies are not (as a rule) used to prevent double counting; that is done by determinants. Generally a Regular Dimension is based on Data Source Query Subjects (i.e. it is a Model Regular Dimension). The determinants in the underlying Data Source Query Subjects prevent double counting in Model Regular Dimensions Therefore, for Model Regular Dimensions, the underlying Query Subjects should have determinants declared. Model vs. Data Source Regular Dimensions You can convert the Generated SQL of a Model Regular Dimension to create a Data Source Regular Dimension. When you do that, there are no longer any underlying Data Source Query Subjects and therefore no determinants In this case, FM uses the dimensional hierarchy information (just like FM 1.x) to prevent double counting Because it behaves like 1.x, you should only specify one hierarchy in a Data Source Regular Dimension Compare Query Subject to Measure Dimension Fact Query Subject (reporting) Measure Dimension (OLAP) Compare Relationships to Scope Relationships Query Subject Relationship diagram Scope Relationship diagram Workshop 1: (Task 7- 11) Dimensional Model Create dimension objects for simple Sales Analysis Product Order Method Time Sales Product Forecast Publish the information Happy Reporting Appendix A Going Forward Regardless of how many layers one makes to present metadata to the business users. We still require a dimensional aware model to present or end-users with the most accurate and predictable results. Here are some of the approaches (Scenarios) we see going forward, the point is all are the correct method of providing business information to the end-users. This demonstrates the flexibility and ease of use of Framework Manager. Dimensionally modeled relational Upgrading a best practice model from CRN 1.1 Maintaining a CRN 1.1 model options Options for modeling normalized databases Dimensional Modeling of Relational Data Sources You must dimensionally model a relational data source when you want to do one or more of the following: use Analysis Studio enable drill functionality in reports improve control over query granularity access member functions in the report authoring tools We recommend that you follow this workflow when you dimensionally model relational data sources: Import the metadata. Check the imported metadata Customize the metadata. Simplify the model using dimensional concepts Resolve ambiguous relationships Define dimensions and determinants Organize the model by creating business views. Create star schema groups Apply security. Create packages and publish metadata. Checking Imported Metadata After importing metadata, you must check the imported metadata in these areas: relationships and cardinality the Usage property for query items the Regular Aggregate property for query items Cardinality is combined with dimensions to control how queries are generated so that you can prevent double-counting automatically resolve loop joins enable cross-fact querying for reporting and analysis Detecting Cardinality from the Data Source When importing from a relational data source, cardinality is detected based on a set of rules that you specify. The available options are use primary and foreign keys use matching query item names that represent uniquely indexed columns use matching query item names The most common situation is to use primary and foreign keys as well as matching query items that represent uniquely indexed columns. The information is used to set some properties of query items as well as to generate relationships. Cardinality in Generated Queries Framework Manager supports both minimum-maximum cardinality and mandatory-optional cardinality. 0:1 - 0 is the minimum cardinality, 1 is the maximum cardinality 1:n - 1 is the minimum cardinality, n is the maximum cardinality A relationship with cardinality 1-n can be specified as 1:1 to 1:n when reading the maximum cardinalities. The minimum cardinality of 1 is set to 0 for optional data. Therefore a 1-n relationship can also be specified as 0:1 to 0:n, 0:1 to 1:n, or 1:1 to 0:n. The basic rules for how cardinality is used are the following: Cardinality is applied in the context of a query, meaning that only the cardinalities of items explicitly included in the query are evaluated. 1-n cardinality implies fact on the n side and implies dimension on the 1 side. A query subject can behave as a dimension in one query and as a fact in another. Queries on more than one fact will result in a stitched query. Modeling 1-n Relationships as 1-1 Relationships If a 1-n relationship exists in the data but is modeled as a 1-1 relationship, SQL traps cannot be avoided because the information provided by the metadata to the query engine is insufficient. The most common problems that arise if 1-n relationships are modeled as 1-1 are the following: Double-counting for multiple-grain queries is not automatically prevented. Cognos 8 cannot detect facts and then generate a stitched query to compensate for double-counting, which can occur when dealing with hierarchical relationships and different levels of granularity across conformed dimensions. Multiple-fact queries are not automatically detected. Cognos 8 does not have sufficient information to detect a multiple-fact query. For multiple-fact queries, an inner join is performed and the loop join is eliminated by dropping the last evaluated join. Dropping a join is likely to lead to incorrect or unpredictable results depending on the dimensions and facts included in the query. You can work around these issues in Report Studio by creating an inner or outer join between queries. This requires the report author to have detailed knowledge of the data. There is no workaround available in Query Studio or Analysis Studio. Analyzing the Model for Facts and Dimensions To dimensionally model a relational data source for the purpose of cross-fact querying and analysis, you must examine the underlying schema and address areas where the role of an object as a fact or a dimension is not clear. This examination can result in creating a logical layer to present the data as a star schema. Or you may decide to make changes to the physical data source to deliver a higher performance application. Physical changes may include consolidating tables by using extract-transform-load tools or creating materialized views for frequently-used aggregations. How schemas can be interpreted This example shows different query subjects. Arrows point to the query subjects whose cardinality indicates that they are always facts. Areas where the behavior is dependent on the context of the query are circled. All other query subjects behave as dimensions in all queries. Simplifying the Model Using Dimensional Concepts These rules apply when modeling with query subjects or dimensions: Create a regular dimension for groups of query subjects that have hierarchical relationships representing a single business concept Create a measure dimension for groups of query subjects that have factual data sharing many regular dimensions or having a master-detail type of relationship Use the cardinality rules to identify areas of the model where the role of an object as fact or dimension is not clear. The result of simplifying the model should be a layer of regular and measure dimensions that clearly represent the data in terms of business concepts and ensure predictable query generation. If your data source contains fact-to-fact relationships, we recommend that you handle these in the data source. Resolving Ambiguous Relationships Ambiguous relationships occur when the data represented by a query subject or dimension can be viewed in more than one context or role. The most common ambiguous relationships are: multiple valid relationships reflexive and recursive relationships You can resolve ambiguous relationships in Framework Manager, or the database administrator can fix them in the data source. For example, the database administrator can remove extra relationships, although this may not be practical. Resolving ambiguous relationships in the model involves determining which query path to use. Multiple Valid Relationships A table with multiple valid relationships between itself and another table is known as a role-playing dimension. Multiple valid relationships often occur between regular dimensions and measure dimensions. This is most commonly seen in dimensions such as Time and Customer. For example, the Sales fact has multiple relationships to the Time dimension on keys such as Order Day, Ship Day, and Close Day. Model Objects vs. Shortcuts You can create model objects based on the original query subjects. These model objects can be model regular dimensions or model query subjects. Or you can create shortcuts. Use model regular dimensions or model query subjects when you want to create custom views of the same query subject. Use shortcuts when you want exact replicas of a regular dimension or query subject in more than one place. Reflexive and Recursive Relationships Reflexive and recursive relationships imply two or more levels of granularity. Framework Manager imports reflexive relationships but does not use them when executing queries. Reflexive relationships, which are self-joins, are shown in the model for the purpose of representation only. To create a functioning reflexive relationship, you must create an alias using either a shortcut to the query subject in the same folder, or a model query subject based on the query subject. You then create a relationship between the query subject and the alias. Using a model query subject tends to be a better option because you can specify which query items are included in the query subject. For example, the Staff query subject has a recursive relationship between Staff Key and Manager Key. Defining Dimensions and Determinants Use dimensions and determinants to ensure that the aggregation in reports is correct and that queries generate correctly. Use regular dimensions to organize and present descriptive information to guide the end user experience in the report authoring tools. Hierarchies are presented in a top-down format, from the coarsest level of granularity to the finest. Data source regular dimensions treat hierarchy and level information as if they are determinants. Model regular dimensions require determinants to be specified for the underlying query subjects. Determinants are used to specify which data is unique. Joins and granularity are the key factors in SQL generation. Specify granularity for the objects that have relationships by using either a hierarchy or determinants on the object to which the join is created. Creating Star Schema Groups Star schema groups can make the model more intuitive for end users by indicating which regular dimensions and measure dimensions are related. Star schema groups can also facilitate multiple-fact reporting by indicating how to use regular dimensions that are common to many measure dimensions. Multiple-fact reporting is also known as cross-functional reporting. Star schema groups also provide context for queries with ambiguous joins. By creating star schema groups in the business view of the model, you can clarify which join path to select when multiple join paths are available. This is particularly useful for fact-less queries. Upgrading a Best Practice Model from ReportNet 1.x If you used the modeling recommendations in the Best Practices for Modeling in Cognos ReportNet 1.x paper, which is available from Cognos on request, to create the ReportNet 1.x model, we recommend the following workflow: Review the existing model Upgrade the model Define dimensions and determinants Create measure dimensions or convert facts to measure dimensions. Create and test star schema groups Republish the metadata.