Post on 16-Apr-2017
transcript
Operationalizing Data Governance through Data Quality Control
David Loshin Knowledge Integrity, Inc.
www.knowledge-integrity.com
1 © 2013 Knowledge Integrity, Inc. www.knowledge-integrity.com
(301)754-6350
Linking Organization to Practice p Mapping business value drivers to
data expectations and objectives, granting oversight and accountability, and verifying performance of compliance with corporate information policies n Processes prescribed for operations, n Procedures for day-to-day
observance n Oversight for verifying compliance
© 2013 Knowledge Integrity, Inc. www.knowledge-integrity.com
(301)754-6350
2
Accountability
Data Policies
Processes & Best Practices
Information Standards
Business Policy
Roles and responsibilities
Program management
Data policies and standards
Business intelligence
Business terminology
Data quality
Auditability
Business Value and Data Dependence
© 2013 Knowledge Integrity, Inc. www.knowledge-integrity.com
(301)754-6350
3
Expenses
Risk Management
Revenue Customer Experience
Performance
p Business policies, corporate mission, and strategic performance objectives can be translated into dimensions of value
p These criteria are used for prioritizing effort in relation to maximizing value
p Data governance helps establish the relationship between value drivers and information utility
Sources of Information Policy
© 2013 Knowledge Integrity, Inc. www.knowledge-integrity.com
(301)754-6350
4
Expenses
Risk Management
Revenue Customer Experience
Performance
• Customer lifetime value analysis
• Voice of the customer
• Satisfaction surveys
• Asset productivity analysis
• Human capital performance
• Regulations • Operational risk • Market factors
• Customer acquisition and retention
• Investment opportunities
• Product performance
• Spend analysis • Commodity risk • Cost management
Data Management Challenges
Data Use and ReUse
Data Requirements
Reinterpreting Semantics
Measurement Triage and Remediation
Impact Assessment
© 2013 Knowledge Integrity, Inc. www.knowledge-integrity.com
(301)754-6350
5
The Data Policy Lifecycle: Actualizing Governance
p Refinement of business requirements for facets of information utility
p Specification for data quality measures and level of acceptability
p Determination of functional requirements to facilitate continuous compliance
© 2013 Knowledge Integrity, Inc. www.knowledge-integrity.com
(301)754-6350
6
Determination of need
Drafting a Policy
Policy Review & Approval
Design & Development Marketing Deployment
Information Policies and Data Governance
Integrated Data
Governance
Manage User Requirements
Data Discovery
Shared Semantics
Embedded Validation
© 2013 Knowledge Integrity, Inc. www.knowledge-integrity.com
(301)754-6350
7
• Business metadata management • Data quality discovery and assessment • Specifying data quality rules • Inspection, monitoring, measurement • Managing data lineage
Managing the Quality of Business Metadata
p Many sources of entity concepts and business terms may conflict with each other
p The data governance framework must facilitate the collection, documentation, and harmonization of business terms
© 2013 Knowledge Integrity, Inc. www.knowledge-integrity.com
(301)754-6350
8
Policies
System Docs
Processes
Models
Standards
Applications
Business Rules
Profiling
Etc.
Entity Concepts
Business Terms
Definition Contextual Meaning
… …
Definition Contextual Meaning
Definition Contextual Meaning
Definition Contextual Meaning
Data Discovery
p Data Discovery enables these types of questions to be answered: n What data sets are available? n What entities are embedded? n What data elements are
available? n How is the data accessed? n What are the quality
constraints? p The results can be shared via
a platform for managing semantic metadata
© 2013 Knowledge Integrity, Inc. www.knowledge-integrity.com
(301)754-6350
9
Data
Integ
ratio
n
Attribute
First d 4 6 y
Last f 6 2 h
Street d 4 7 n
City a 0 2 o
State
Value Count
A 12000
I 10000
L 7655
X 3208
N 120
M 8
Data Quality Assessment
p Analysis of data sets, records, data elements, and data values to n Identify potential anomalies n Determine business impacts n Evaluate dimensions for measurement of data quality
© 2013 Knowledge Integrity, Inc. www.knowledge-integrity.com
(301)754-6350
10
Analysis
Data Quality: Expectations, Rules, and Monitoring
p Data quality rules can be used to monitor conformance to data policies
p Conformance can be measured, thresholded, and reported at each handoff location in the processing stream
p Specific failures can generate events as directed by Data Quality Service Level Agreements
p Static auditing: measurement applied to a “static” data set n Examples: SQL queries, data profiling tools
p Inlined monitoring: measurement performed within a process flow n Example: edit checks, dynamic monitors
p All measurements are compared against acceptability thresholds
p Acceptability threshold is related to the degree of impact
11 11 © 2013 Knowledge Integrity, Inc. www.knowledge-integrity.com
(301)754-6350
Data Quality Rules: Measures and Thresholds
p Provide specific n Measures n Methods of measurement n Units of measures n Levels of acceptability
© 2012 Knowledge Integrity, Inc. www.knowledge-integrity.com
(301)754-6350
12
© 2013 Knowledge Integrity, Inc. www.knowledge-integrity.com
(301)754-6350
13 13
Data Quality Control
p Controls measure observance of data expectations based on information policies and corresponding data rules
p Those rules are refined based on an analysis of the data dependencies and defined expectations
p Controls are placed at relevant locations within the process stream
Producer Process
Consumer Process
As data is handed off between process tasks,
controls validate accuracy, completeness, consistency, timeliness against defined
expectations
Instituting Inspection Using Data Quality Rules
p Apply tools and techniques for measuring conformance to data rules (using data profiling and data monitoring tools):
© 2013 Knowledge Integrity, Inc. www.knowledge-integrity.com
(301)754-6350
14
Instituting Inspection Using Data Quality Rules
p Data quality expectations are inspected and any emerging issues are identified:
© 2013 Knowledge Integrity, Inc. www.knowledge-integrity.com
(301)754-6350
15
Instituting Inspection Using Data Quality Rules
p Different events can be triggered by a data failure, such as notifications to data stewards:
© 2013 Knowledge Integrity, Inc. www.knowledge-integrity.com
(301)754-6350
16
Instituting Inspection Using Data Quality Rules
p Or logging the failure in a Data Quality Incident Management System and score card:
© 2013 Knowledge Integrity, Inc. www.knowledge-integrity.com
(301)754-6350
17
Instituting Inspection Using Data Quality Rules
p Effectiveness demonstrated when: n Control events occur when data failure events take place, n The proper mitigation or remediation actions are performed, n The corrective actions to correct the problem and eliminate its root
cause are performed within a reasonable time frame, and n A control event for the same issue is never triggered further
downstream p Measurements can be aggregated over time into performance
metrics
© 2013 Knowledge Integrity, Inc.
www.knowledge-integrity.com (301)754-6350
18
Integrating Data Quality Reporting with Governance
© 2013 Knowledge Integrity, Inc. www.knowledge-integrity.com
(301)754-6350
19
Processing Stage
Processing Stage
Processing Stage
Processing Stage
Tools & Processes: Operationalizing Data Governance
p Methods and tools for data discovery: profiling data, statistical analysis of values, and model evaluation
p Metadata management through a central platform for knowledge capture and communication
p End-to-end visibility of lineage for structure, semantics, and use across enterprise
p Data Quality assessment p Integrated data quality control p Inspection, monitoring, and reporting
© 2013 Knowledge Integrity, Inc. www.knowledge-integrity.com
(301)754-6350
20
Questions and Open Discussion
p www.knowledge-integrity.com
p If you have questions,
comments, or suggestions, please contact me David Loshin 301-754-6350 loshin@knowledge-integrity.com
© 2013 Knowledge Integrity, Inc. www.knowledge-integrity.com
(301)754-6350
21
www.dataqualitybook.com
www.mdmbook.com