Statistical Distortion: Consequences of Data Cleaning
Data Cleaning & IntegrationCompSci 590.01 Spring 2017
Junyang Gao, Amir Rahimzadeh Ilkhechi, Yuhao Wen
Some contents were based on :Tamraparni Dasu’s DSAA Tutorial, 2016
Tamraparni Dasu, Ji Meng Loh. “Statistical Distortion: Consequences of Data Cleaning.” VLDB, 2012
Biggest take-away points?
(For us:)
● Cleaner data do not necessarily imply more useful or useable data
● In practice, simple cleaning strategy may outperform a more sophisticated method that have assumptions not suitable over the data
Outline
● Introduction & Experimental Framework● Methodology & Formulation● Experiments & Analysis
Data Cleaning or Data Mangling?● Changed the shape:
a. Most frequent values (Mode)
b. Least frequent values (Anomalies)
● Moved good values● Turned good values to
glitches
How to measure data cleaning strategies?
● Three dimensional data quality metric:a. Statistical Distortionb. Glitch Improvementc. Cost
Experimental Framework
1. Glitch Index● Weighted Sum:
● The lower the glitch index,
The “cleaner” the data set
Name City State
0 0 0
0 0 0
0 0 1
0 0 1
0 0 1
0 0 0
0 0 0
0 0 0
0 0 0
Glitch Vector
Experimental Framework
2. Statistical Distortion
Distance between two distributions
1. Kullback-Liebler “distance”● P,Q are two probability distributions over the same event
space●
Distance between two distributions
1. Kullback-Liebler divergence●
Entropy of PCross-Entropy of P,Q
Distance between two distributions
2. Jensen–Shannon divergence
● symmetrized and smoothed version of the Kullback–Leibler divergence
● , where
Distance between two distributions
3. Earth Mover’s distance
● Minimum cost of converting P to Q, transportation problem●
Experimental Framework
3. Cost
● Highly context dependent● In this paper: glitch-based (percentage of glitches
removed)
Experimental Framework
“Best” strategies depends on user’s tolerance
● Statistical Distortion● Glitch Improvement● Cost
Outline
● Introduction & Experimental Framework● Methodology & Formulation● Experiments & Analysis
Applicable to: ● Structured,● Hierarchical,● Spatio-temporal,● Unstructured data
A hierarchical network exampleN_1
N_13
N_132
A hierarchical network example (cont.)
● Each node measures v variables (time series)● For N_ijk: represents the collected data at time t● F_t: the history up to time t-1● represents the window of time-step history from t-⍵ up
to time t-1
Glitch Types:● Multitype
● Co-occurring
● Stand alone
Glitch DetectionGlitch detector is a function of
X^t
Missing values
Glitch Detection(cont.)
Glitch detector is a function of X^t
Inconsistent values
Glitch Detection(cont.)
Glitch detector is also a function of other parameters
Outlier values
Glitch Detection(cont.)
Glitch Matrix:
Glitch Index*
*1 ⨉ p is a typo in the paper
Statistical Distortion Measure(EMD)*
Slides adopted from Pete Barnum presentation*
Statistical Distortion Measure(EMD)
Statistical Distortion Measure(EMD)
Linear programming approach for EMD:
Linear programming approach for EMD:
Linear programming approach for EMD:
Linear programming approach for EMD:
Linear programming approach for EMD (as an instance of transportation problem):
Linear programming approach for EMD (constraint 1):
Linear programming approach for EMD (constraint 2):
Linear programming approach for EMD (constraint 3):
Linear programming approach for EMD (constraint 4):
Final result for EMD:
Final result for EMD:
Outline
● Introduction & Experimental Framework● Methodology & Formulation● Experiments & Analysis
ExperimentDataset:
● 20,000 time series , Length at most 170, 3 variables
Glitches
● Inconsistencies○ A1 >= 0○ 0<= A3 <=1○ If A3 is missing, A1 should not be populated
● Outliers● Missing values
All graph in the following slides are from T. Dasu and J. Loh. "Statistical Distortion: Consequences of Data Cleaning." VLDB 2012.
ExperimentSampling
● From D & DI with replacement, 50 pairs in total● Sample size: 100, 500 (no significant impact)
Factors concerned:
● Attribute transformations● Strategies● Cost● Sample size
Experiment
Analysis
● Cleaning Strategies● Studying Cost● Data Transformation and Cleaning● Strategies and Attribute Distributions● Strategies Evaluation● Cleaning Cost
Analysis - Cleaning StrategiesApplied 5 strategies to each of the 100 test pairs of data streams.
Strategies Missing values Inconsistent values Outliers
1 Impute using SAS PROC MI Winsorization by attribute basis
2 Impute using SAS PROC MI ignore
3 ignore ignore Winsorization by attribute basis
4 Replace with mean attribute from ideal dataset ignore
5 Replace with mean attribute from ideal dataset & Winsorization (outlier only)
weight 0.25 0.25 0.5
Analysis - Studying Cost
● Cost ~ Proportion of the glitches cleaned● Process:
○ Compute normalized glitch score for each time series○ Rank them○ Top x% cleaned (x=0 -> Nothing cleaned, x=100 -> everything cleaned)
Analysis - Data Transformation and CleaningStrategy: 1
Gray:
Imputed missing values
X=Y:
Untouched data
Back dots:
Winsorized values
Analysis - Strategies and Attribute Distributions
Analysis - Strategies Evaluation
Figure 6
Analysis - Strategies Evaluation
● Single cleaning method○ SAS PROC MI○ Mean
● Winsorization only● Using two methods
○ Impute+ Winsorize○ Mean + Winsorize
Analysis - Cleaning Cost
Strategy: Imputation + Winsorization
