DWH MATERIAL Business Intelligence Study Guide Business Intelligence (BI): Business Intelligence refers to a set of methods and techniques that are used by organizations for tactical and strategic decision making. Data Warehousing : Integration of data from multiple sources into large warehouses and support of on-line analytical processing and business decision making DW vs. Operational Databases Data Warehouse Subject Oriented Integrated Nonvolatile Time variant Ad hoc retrieval Operational Databases Application oriented Limited integration Continuously updated Current data values only Predictable retrieval Data Warehouse: a subject-oriented, integrated, time- variant , and nonvolatile collection of data in support of management’s decision-making process. Data Mart A monothematic data warehouse Department- oriented or business line oriented It is a subset of Data Warehouse which is department level Praveen 9535696199 Page 1
Transcript
DWH MATERIAL
Business Intelligence Study Guide
Business Intelligence (BI): Business Intelligence refers to a set of methods and techniques that are used by organizations for tactical and strategic decision making.
Data Warehousing: Integration of data from multiple sources into large warehouses and support of on-line analytical processing and business decision making
DW vs. Operational Databases
Data Warehouse Subject Oriented Integrated Nonvolatile Time variant Ad hoc retrieval
Operational Databases Application oriented Limited integration Continuously updated Current data values only Predictable retrieval
Data Warehouse: a subject-oriented, integrated, time-variant, and nonvolatile collection of data in support of management’s decision-making process.
Data Mart A monothematic data warehouse Department- oriented or business line oriented It is a subset of Data Warehouse which is department level
Praveen 9535696199 Page 1
DWH MATERIAL
Difference between data mart and data warehouse
Data Mart Data Warehouse
Data mart is usually sponsored at the department level and developed with a specific issue or subject in mind, a data mart is a data warehouse with a focused objective.
Data warehouse is a “Subject-Oriented, Integrated, Time-Variant, Nonvolatile collection of data in support of decision making”.
A data mart is used on a business division/ department level.
A data warehouse is used on an enterprise level
A Data Mart is a subset of data from a Data Warehouse. Data Marts are built for specific user groups.
A Data Warehouse is simply an integrated consolidation of data from a variety of sources that is specially designed to support strategic and tactical decision making.
By providing decision makers with only a subset of data from the Data Warehouse, Privacy, Performance and Clarity Objectives can be attained.
The main objective of Data Warehouse is to provide an integrated environment and coherent picture of the business at a point in time.
Top-Down Approach (Bill Inmon)
Praveen 9535696199 Page 2
DWH MATERIAL
Advantages • A truly corporate effort, an enterprise view of data• Inherently architected – not a union of disparate data marts• Single, central storage of data about the content• Centralized rules and control• May see quick results if implemented with iterations
Disadvantages • Takes longer to build even with an iterative method• High exposure/risk to failure• Needs high level of cross-functional skills• High outlay without proof of concept
Bottom-Up Approach (Ralph Kimball)
Advantages• Faster and easier implementation of manageable pieces• Favorable return on investment and proof of concept• Less risk of failure• Inherently incremental; can schedule important data marts first• Allows project team to learn and grow
Disadvantages • Each data mart has its own narrow view of data• Permeates redundant data in every data mart• Perpetuates inconsistent and irreconcilable data
Praveen 9535696199 Page 3
DWH MATERIAL
• Proliferates unmanageable interfaces
Data Staging Component
• Three major functions need to be performed for getting the data ready (ETL)– extract the data– transform the data – and then load the data into the data warehouse storage
Data Warehouse
– Subject-Oriented - Data is stored by subjects
– Integrated Data - Need to pull together all the relevant data from the various systems
• Data from internal operational systems• Data from outside sources
– Time-Variant Data - the stored data contains the current values• The use needs data not only about the current purchase, but on the
past purchases
– Nonvolatile Data - Data from the operational systems are moved into the data warehouse at specific intervals
– Data Granularity - Data granularity in a data warehouse refers to the level of detail
• The lower the level of detail, the finer the data granularity• The lowest level of detail ® a lot of data in the data warehouse
Important features of a DWH
DRILL DOWN, DRILL ACROSS, Graphs, PI charts, dashboards and TIME HANDLING
To be able to drill down/drill across is the most basic requirement of an end user in a data warehouse.
Praveen 9535696199 Page 4
DWH MATERIAL
Schema • Graphical Representation of the data structure.• First Phase in implementation of Universe
Star schema• Fact tables contain factual or quantitative data• 1:N relationship between dimension tables and fact tables • Dimension tables contain descriptions about the subjects of the business • Dimension tables are denormalized to maximize performance
Praveen 9535696199 Page 5
DWH MATERIAL
Snow flake schema• Unlike Star-Schema, Snowflake schema contain normalized dimension tables in a
tree like structure with many nesting levels.• Snowflake schema is easier to maintain but queries require more joins.
Praveen 9535696199 Page 6
DWH MATERIAL
Granularity
Principle: Create fact tables with the most granular data possible to support analysis of the business process.
In Data warehousing grain refers to the level of detail available in a given fact table as well as to the level of detail provided by a star schema.
It is usually given as the number of records per key within the table. In general, the grain of the fact table is the grain of the star schema.
Grain of the star schema
In Data warehousing grain refers to the level of detail available in a given fact table as well as to the level of detail provided by a star schema.
It is usually given as the number of records per key within the table. In general, the grain of the fact table is the grain of the star schema.
Praveen 9535696199 Page 7
DWH MATERIAL
Fact Table
A Fact Table in a dimensional model consists of one or more numeric facts of importance to a business.
Examples of facts are as follows:
the number of products sold
the value of products sold
the number of products produced
the number of service calls received
Fact and Dimension
A "fact" is a numeric value that a business wishes to count or sum. A "dimension" is essentially an entry point for getting at the facts. Dimensions are
things of interest to the business.
Types of facts
Additive: Additive facts are facts that can be summed up through all of the dimensions in the fact table.
Semi-Additive: Semi-additive facts are facts that can be summed up for some of the dimensions in the fact table, but not the others.
Non-Additive: Non-additive facts are facts that cannot be summed up for any of the dimensions present in the fact table.
Factless Fact Table
Factless fact table captures the many-to-many relationships between dimensions, but contains no numeric or textual facts. They are often used to record events or coverage information.
Common examples of Factless fact tables include: Identifying product promotion events (to determine promoted products that
didn’t sell)
Tracking student attendance or registration events
Tracking insurance-related accident events
Praveen 9535696199 Page 8
DWH MATERIAL
Conformed Dimension
These dimensions are something that is built once in your model and can be reused multiple times with different fact tables.
For example, consider a model containing multiple fact tables, representing different data marts.
Slowly changing dimensions• Are the Customer and Product Dim independent of Time Dim?• Changes in names, family status, product district/region• How to handle these changes in order not to affect the history status? Eg.
Insurance• 3 suggestions for slowly changing dimensions
Type 1: The new record replaces the original record. No trace of the old record exists.
Type 2: A new record is added into the customer dimension table. Therefore, the customer is treated essentially as two people.
Type 3: The original record is modified to reflect the change.
Type 1 Slowly Changing Dimension
In Type 1 Slowly Changing Dimension, the new information simply overwrites the original information. In other words, no history is kept.
In our example, recall we originally have the following table:
Customer Key Name State
1001 Christina Illinois
After Christina moved from Illinois to California, the new information replaces the new record, and we have the following table:
Customer Key Name State
1001 Christina California
Advantages:
- This is the easiest way to handle the Slowly Changing Dimension problem, since there is no need to keep track of the old information.
- All history is lost. By applying this methodology, it is not possible to trace back in history. For example, in this case, the company would not be able to know that Christina lived in Illinois before.
Usage:
About 50% of the time.
When to use Type 1:
Type 1 slowly changing dimension should be used when it is not necessary for the data warehouse to keep track of historical changes.
Type 2 Slowly Changing Dimension
In Type 2 Slowly Changing Dimension, a new record is added to the table to represent the new information. Therefore, both the original and the new record will be present. The new record gets its own primary key.
In our example, recall we originally have the following table:
Customer Key Name State
1001 Christina Illinois
After Christina moved from Illinois to California, we add the new information as a new row into the table:
Customer Key Name State
1001 Christina Illinois
1005 Christina California
Advantages:
- This allows us to accurately keep all historical information.
Disadvantages:
- This will cause the size of the table to grow fast. In cases where the number of rows for the table is very high to start with, storage and performance can become a concern.
- This necessarily complicates the ETL process.
Usage:
Praveen 9535696199 Page 10
DWH MATERIAL
About 50% of the time.
When to use Type 2:
Type 2 slowly changing dimension should be used when it is necessary for the data warehouse to track historical changes.
Type 3 Slowly Changing Dimension
In Type 3 Slowly Changing Dimension, there will be two columns to indicate the particular attribute of interest, one indicating the original value, and one indicating the current value. There will also be a column that indicates when the current value becomes active.
In our example, recall we originally have the following table:
Customer Key Name State
1001 Christina Illinois
To accomodate Type 3 Slowly Changing Dimension, we will now have the following columns:
Customer Key Name Original State Current State Effective Date
After Christina moved from Illinois to California, the original information gets updated, and we have the following table (assuming the effective date of change is January 15, 2003):
Customer Key Name Original State Current State Effective Date
1001 Christina Illinois California 15-JAN-2003
Advantages: - This does not increase the size of the table, since new information is updated. - This allows us to keep some part of history. Disadvantages: - Type 3 will not be able to keep all history where an attribute is changed more than once. For example, if Christina later moves to Texas on December 15, 2003, the California information will be lost. Usage: Type 3 is rarely used in actual practice. When to use Type 3: Type III slowly changing dimension should only be used when it is necessary for the data warehouse to track historical changes, and when such changes will only occur for a finite number of time.
Praveen 9535696199 Page 11
DWH MATERIAL
Junk Dimensions
A "junk" dimension is a collection of random transactional codes, flags and/or text attributes that are unrelated to any particular dimension. The junk dimension is simply a structure that provides a convenient place to store the junk attributes.
A good example would be a trade fact in a company that brokers equity trades.
• Leave the flags in the fact tables– likely sparse data– no real browse entry capability– can significantly increase the size of the fact table
• Remove the attributes from the design– potentially critical information will be lost– if they provide no relevance, remove them
• Make a flag into it’s own dimension– may greatly increase the number of dimensions, increasing the size of the
fact table– can clutter and confuse the design
• Combine all relevant flags, etc. into a single dimension– the number of possibilities remain finite– information is retained
Degenerated Dimension
• An item that is in the fact table but is stripped off of its description, because the description belongs in dimension table, is referred to as Degenerated Dimension. Since it looks like dimension, but is really in fact table and has been degenerated of its description, hence is called degenerated dimension.
• A dimension which is located in fact table is called degenerated dimension.
Dimensional Modeling
• A type of data modeling suited for data warehousing.• In a dimensional model, there are two types of tables: dimensional tables and fact
tables.• Dimensional table records information on each dimension and fact table records
all the fact or measure.
Praveen 9535696199 Page 12
DWH MATERIAL
Four steps in Dimensional modeling1. Identify the process being modeled.2. Determine the grain at which facts will be stored.3. Choose the dimensions.4. Identify the numeric measures for the facts.
There are three levels of data modeling. They are conceptual, logical, and physical. This section will explain the difference among the three, the order with which
each one is created, and how to go from one level to the other.
Conceptual Data Model
Features of conceptual data model include: Includes the important entities and the relationships among them.
No attribute is specified.
No primary key is specified.
At this level, the data modeler attempts to identify the highest-level relationships among the different entities.
Logical Data Model Features of logical data model include:
Includes all entities and relationships among them.
Foreign keys (keys identifying the relationship between different entities) are specified.
Normalization occurs at this level.
At this level, the data modeler attempts to describe the data in as much detail as possible, without regard to how they will be physically implemented in the database. In data warehousing, it is common for the conceptual data model and the logical data model to be combined into a single step (deliverable). The steps for designing the logical data model are as follows:
1. Identify all entities.
2. Specify primary keys for all entities.
3. Find the relationships between different entities.
4. Find all attributes for each entity.
5. Resolve many-to-many relationships.
6. Normalization.
Physical Data Model Features of physical data model include:
Specification all tables and columns.
Foreign keys are used to identify relationships between tables.
Demoralization may occur based on user requirements.
Physical considerations may cause the physical data model to be quite different from the logical data model.
At this level, the data modeler will specify how the logical data model will be realized in the database schema.
The steps for physical data model design are as follows: 1. Convert entities into tables.
2. Modeling is an efficient and effective way to represent the organizations needs. It provides information in a graphical way to the members of an organization to understand and communicate the business rules and processes. Business Modeling and Data Modeling are the two important types of modeling.
Logical Data Model Physical Data Model
Represents business information and defines business rules
Represents the physical implementation of the model in a database.
Entity Table
Attribute Column
Primary Key Primary Key Constraint
Alternate Key Unique Constraint or Unique Index
Inversion Key Entry Non Unique Index
Rule Check Constraint, Default Value
Relationship Foreign Key
Definition Comment
3 types of multidimensional data– Data from external sources (represented by the blue cylinder) is copied
into the small red marble cube, which represents input multidimensional data
– Pre-calculated, stored results derived from it – on-the-fly results, calculated as required at run-time, but not stored in a
database–
What is Staging area why we need it in DWH?
If target and source databases are different and target table volume is high it contains some millions of records in this scenario without staging table we need to design your informatica using look up to find out whether the record exists or not in the target table since target has huge volumes so its costly to create cache it will hit the performance.
Praveen 9535696199 Page 15
DWH MATERIAL
If we create staging tables in the target database we can simply do outer join in the source qualifier to determine insert/update this approach will give you good performance.
It will avoid full table scan to determine insert/updates on target.And also we can create index on staging tables since these tables were designed for specific application it will not impact to any other schemas/users.
While processing flat files to data warehousing we can perform cleansing. Data cleansing, also known as data scrubbing, is the process of ensuring that a set of data is correct and accurate. During data cleansing, records are checked for accuracy and consistency.
Since it is one-to-one mapping from ODS to staging we do truncate and reload.
We can create indexes in the staging state, to perform our source qualifier best.
If we have the staging area no need to relay on the informatics transformation to known whether the record exists or not.
Data cleansing
Weeding out unnecessary or unwanted things (characters and spaces etc) from incoming data to make it more meaningful and informative
Data merging
Data can be gathered from heterogeneous systems and put together
Data scrubbing
Data scrubbing is the process of fixing or eliminating individual pieces of data that are incorrect, incomplete or duplicated before the data is passed to end user.
Data scrubbing is aimed at more than eliminating errors and redundancy. The goal is also to bring consistency to various data sets that may have been created with different, incompatible business rules.
ODS (Operational Data Sources):
It is a replica of OLTP system to reduce the burden on production system (OLTP) while fetching data for loading targets. Hence it is a mandate Requirement for every Warehouse.
OLTP is a sensitive database they should not allow multiple select statements it may impact the performance as well as if something goes wrong while fetching data from OLTP to data warehouse it will directly impact the business.ODS is the replication of OLTP.ODS is usually getting refreshed through some oracle jobs.It enables management to gain a consistent picture of the business.
Surrogate key
A surrogate key is a substitution for the natural primary key. It is a unique identifier or number (normally created by a database sequence
generator) for each record of a dimension table that can be used for the primary key to the table.
A surrogate key is useful because natural keys may change.
Aggregation
– The system uses physically stored aggregates as a way to enhance performance of common queries.
– These aggregates, like indexes, are chosen silently by the database if they are physically present.
– End users and application developers do not need to know what aggregates are available at any point in time, and applications are not required to explicitly code the name of an aggregate
When you go for higher level of aggregates, the sparsity percentage goes down, eventually reaching 100% of occupancy
Data Quality Issues• Dummy values in fields• Missing data• Unofficial use of fields• Cryptic values• Contradicting values• Reused primary keys• Inconsistent values• Incorrect values• Multipurpose fields
Praveen 9535696199 Page 17
DWH MATERIAL
Steps in Data Cleansing Parsing Correcting Standardizing Matching Consolidating
OLAP defined:– On-line Analytical Processing(OLAP) is a category of software
technology that enables analysts, managers and executives to gain insight into data through fast, consistent, interactive access in a wide variety of possible views of information that has been transformed from raw data to reflect the real dimensionality of the enterprise as understood by the user
– Users need the ability to perform multidimensional analysis with complex calculations
The basic virtues of OLAP– Enables analysts, executives, and managers to gain useful insights from
the presentation of data– Can reorganize metrics along several dimensions and allow data to be
viewed from different perspectives– Supports multidimensional analysis– Is able to drill down or roll up within each dimension
Praveen 9535696199 Page 18
DWH MATERIAL
BUSINESS METADATA
• Is like a roadmap or an easy-to-use information directory showing the contents and how to get it
• How can I sign onto and connect with the data warehouse?• Which parts of the data warehouse can I access?• Can I see all the attributes from a specific table?• What are the definitions of the attributes I need in my query?• Are there any queries and reports already predefined to give the results I need?
TECHNICAL METADATA• Technical metadata is meant for the IT staff responsible for the development and
administration of the data warehouse
Praveen 9535696199 Page 19
DWH MATERIAL
• Technical metadata is like a support guide for the IT professionals to build, maintain, and administer the data warehouse
Physical Design Objectives• Improve Performance
– In OLTP, 1-2 secs max; in DW secs to mins • Ensure scalability• Manage storage• Provide Ease of Administration• Design for Flexibility.
Partitioning• Breaking data into several physical units that can be handled separately• Not a question of whether to do it in data warehouses but how to do it• Granularity and partitioning are key to effective implementation of a warehouse• Partitions are spread across multiple disks to boost performanceWhy Partition?• Flexibility in managing data• Smaller physical units allow
Criterion for Partitioning• Vertically (groups of selected columns together. More typical in dimension
tables)• Horizontally (e.g. recent events and past history. Typical in fact tables)
Parallelization• The argument goes:
– if your main problem is that your queries run too slowly, use more than one machine at a time to make them run faster (Parallel Processing).
– Oracle uses this strategy in its warehousing products.
Praveen 9535696199 Page 20
DWH MATERIAL
Indexing• Structure separate from the table data it refers to, storing the location of rows in
the database based on the column values specified when the index is created.• They are used in data warehouse to improve warehouse throughput• Indexing and loading• Indexing for large tables
Bitmap Index• Bitmap indices are a special type of index designed for efficient querying on
multiple keys• Records in a relation are assumed to be numbered sequentially from, say, 0
– Given a number n it must be easy to retrieve record n• Particularly easy if records are of fixed size
• Applicable on attributes that take on a relatively small number of distinct values– E.g. gender, country, state, …– E.g. income-level (income broken up into a small number of levels such
as 0-9999, 10000-19999, 20000-50000, 50000- infinity)• A bitmap is simply an array of bits
Praveen 9535696199 Page 21
DWH MATERIAL
• In its simplest form a bitmap index on an attribute has a bitmap for each value of the attribute
– Bitmap has as many bits as records– In a bitmap for value v, the bit for a record is 1 if the record has the value
v for the attribute, and is 0 otherwise
Clustering• The technique involves placing and managing related units of data to be retrieved
in the same physical block of storage
• This arrangement causes related units of data to be retrieved together in one single operation
• In a clustering index, the order of the rows is close to the index order. Close means that physical records containing rows will not have to be accessed more than one time if the index is accessed sequentially
DW Deployment• Major deployment activities
– Complete user acceptance– Perform initial loads– Get user desktops ready– Complete initial user training– Institute initial user support– Deploy in stages–
DW Growth & Maintenance• Monitoring the DW• Collection of Stats• Usage of Stats
– For growth planning– For fine tuning
• User training– Data Content– Applications & Tools
Praveen 9535696199 Page 22
DWH MATERIAL
Dimensional Modeling Exercise
Exercise: Create a star schema diagram that will enable FIT-WORLD GYM INC. to analyze their revenue.
− The fact table will include: for every instance of revenue taken – attribute(s) useful for analyzing revenue.
− The star schema will include all dimensions that can be useful for analyzing revenue.
− The only data sources available are shown bellow.
SOURCE 1 “FIT-WORLD GYM” Operational Database: ER-Diagram and the tables based on it (with data)
