1
iWay DQC and iDP
Kam WongSolutions Architect
Exploring Techniques of Data Quality and Profiling
April 20, 2012
What Is Data Profiling? What Are Some of Data Profiling Techniques? How To Monitor Your Data?
What Is Data Profiling?
Data profiling is about knowing your data
It discovers relationship between data elements, whether they are in the same data source or across multiple, heterogeneous data sources.
It performs statistical analysis against individual columns (as in relational database) discovering such things as the number of null values, patterns, whether the data matches the expected data type and so on.
2
What Are Some Of Data Profiling Techniques?
3
Profiling – Technical Basic Analysis
Minimums Maximums Averages Counts Etc.
Patterns / Masking Domain Extremes Quantities Frequency Analysis Foreign Key Analysis Charting Grouping / Aggregate Drilldown / Interactive Displays
How To Monitor Your Data?
4
View Profiles Compare data
quality over time – trend analysis
Monitor data quality index based on business rules
iWay Data Quality Management Life-Cycle
5
6
Demonstration
Thank-You
7