1

DATA QUALITYThe general method

Data model

Non-conform data

Corrected data / improved IS

Corrected programs Exceptions management

measure

correct

prevent

2

MEASURE DATA QUALITY

DB

Dataacquisition

schema

?

??

?

Treatment

?

Extraction system

?

?

??

The data model is the central point for all actions

objectives questions what to measure

The data contained in databases are the result of a processing

Does the processes (collection, calculation, extraction) respect the structures, relations and data rules?

Data compliance with the data model

The data must allow the users to process tasks

Does the application meet the users requirements ?

Compliance of data model with users requirements

3

MEASURE DATA QUALITY

data

programs

data qualitymodel A

0-1

0-N

avenants-s inis tres

0-N

0-1

contrats -avenants

avenants

s inis tres

contrats

DB

data

programs

data qualitymodel A

0-1

0-N

avenants-s inis tres

0-N

0-1

contrats -avenants

avenants

s inis tres

contrats

DB

application B

information system quality

the organisation model(A+B+ functional links)

0-1

0-N

avenants-s inis tres

0-N

0-1

contrats-avenants

avenants

s inis tres

contrats

0-N

0-N

contrats-s inis tres

0-N

0-1

contrats-avenants

avenants sinis tres

contrats

organistion information system quality

application A

information system quality

real world

4

TO MEASURE DATA QUALITY

5

TO MEASURE DATA QUALITY

6

TO MEASURE DATA QUALITY

7

DATA QUALITYThe general method

Data model

Non-conform data

Corrected data / improved IS

Corrected programs Exceptions management

measure

correct

prevent

8

TO CORRECT

For the data

Concept inadequacy

Fields segmentation and normalization

Fields value cleaning orphan data detection

Occurrences deduplication

For the Information system

Data model and application improvements

9

TO CORRECT

10

TO PREVENT

The deployment of the data quality process must allow :

To clean up the bottom of the river punctually To dam up the arrival of new information flows of doubtful

quality

11

DATA QUALITYThe general method

Data model

Non-conform data

Corrected data / improved IS

Corrected programs Exceptions management

measure

correct

prevent

12

TO PREVENT

Objective :to (re)organize the data flows in order to guarantee a given quality level , so to minimize the corrective process.

Principle : data are products coming from a production line. For this reason, one should apply the quality control principles applied in the industry.

measure at different spots validation referenced with external world measures …

Involved the organization (management, administrative process) as well as technology

People and organisation resistance are important to consider

13

TO PREVENT

Technical issue Program correctionCorrection des programmes Data dictionary consolidation (complete méta-data) DB re-engineering

Organizational issue Identification of the processes and data flows Identification of the critical points and the

responsabilities Users training Organizational restructuring : flow

14

SYNTHESIS

The added value of the proposed approach

Dataprofiling

Reverse-engineering

Rulesdefinition

Datamerge

Programscorrection

Modelevolution

Datadictionary

Exceptionsmanagement

Conceptsprecision

The data quality steps according to Gartnerdata

profilingstandar-disation

deduplication

cleaning follow upenrichment

measure correct prevent

LogicalData

extraction

« Orphan » data

detection

15

Synthesis

What needs to be done

1

measure correct prevent

1 How to guarantee the conformity of the data after an IS merge ?

2 How to manage « old IS data » with respect with the new data management rules ?

3 How to manage the quality of the data flow entering or leaving the IS ?

4 How to manage the data rules with respect to the applications ?

Dataprofiling

Reverse-engineering

To specifyand complete

the rules

To manage the

data dictionary

To specifyand completethe concepts

To correct the data

2 Dataprofiling

Reverse-engineering

To specifyand complete

the rules

To specifyand completethe concepts

To correct the data

To manage the

data dictionary

3To manage

the exceptions

To correct the programs

Reverse-engineering

To specifyand complete

the rules

To specifyand completethe concepts

4 Reverse-engineering

To specify and complete

the rules

To specifyand completethe concepts

To manage the

exceptions

To correct the programs