Copyright © 2016 by Luc Anselin, All Rights Reserved
Luc Anselin
Global Spatial AutocorrelationClustering
http://spatial.uchicago.edu
Copyright © 2016 by Luc Anselin, All Rights Reserved
• join count statistics
• Moran’s I
• Moran scatter plot
• non-parametric spatial autocorrelation
Copyright © 2016 by Luc Anselin, All Rights Reserved
Join Count Statistics
Copyright © 2016 by Luc Anselin, All Rights Reserved
• Recap - Spatial Autocorrelation Statistic
• combination of attribute similarity and locational similarity
• f(xi, xj) and wij
• general form ΣiΣj f(xi,xj)wij
Copyright © 2016 by Luc Anselin, All Rights Reserved
• Spatial Autocorrelation for Binary Data
• only two values observed, e.g., presence and absence
• xi = 1, or B(lack)
• xi = 0, or W(hite)
• coding of 1 or 0 is arbitrary, typically choose 1 for the smaller category
• unlike event analysis (point patterns) all locations are observed with either 1 or 0 values
Copyright © 2016 by Luc Anselin, All Rights Reserved
neighborhoods with burglaries (=1)Walnut Hills, Cincinnati, OH
Copyright © 2016 by Luc Anselin, All Rights Reserved
• Match Value-Location
• count the number of times similar values occur for neighbors = joins
• if xi = 1 and xj = 1 for i-j neighbors, count as a BB join
• if xi = 0 and xj = 0 for i-j neighbors, count as a WW join
Copyright © 2016 by Luc Anselin, All Rights Reserved
• Positive Spatial Autocorrelation
• similarity of neighbors
• BB = 1 -- 1
• xixj = 1 for a join, 0 otherwise
• WW = 0 -- 0
• (1 - xi)(1 - xj) = 1 for a join, 0 otherwise
Copyright © 2016 by Luc Anselin, All Rights Reserved
• Negative Spatial Autocorrelation
• dissimilarity of neighbors
• BW = 1 -- 0 or 0 -- 1
• (xi - xj)2 = 1 for dissimilar neighbors 0 otherwise
Copyright © 2016 by Luc Anselin, All Rights Reserved
• Join Count Statistics
• binary spatial weights, wij = 1 or 0
• BB: (1/2) ΣiΣj wij.xi.xj
• WW: (1/2) ΣiΣj wij.(1-xi).(1-xj)
• BW: (1/2) ΣiΣj wij.(xi - xj)2
• BB + WW + BW = (1/2) S0 = (1/2) ΣiΣj wij
Copyright © 2016 by Luc Anselin, All Rights Reserved
null hypothesis: spatial randomness
Copyright © 2016 by Luc Anselin, All Rights Reserved
Actual Random 1 Random 2
BB 130 95 79
WW 845 808 814
BW 509 581 591
Sum 1484 1484 1484
join count statistics - queen contiguityn = 457, S0 = 2968
Copyright © 2016 by Luc Anselin, All Rights Reserved
• Analytical Inference
• based on sampling theory
• binomial distribution, using sampling with replacement or without replacement
• moments (mean, variance, higher) can be computed
• uses a (poor) normal approximation
• Walnut Hills burglary BB = 130
• E[BB] = 91.7, Var[BB] = 77.9
• z-value = 4.34
• p-value < 0.00007
Copyright © 2016 by Luc Anselin, All Rights Reserved
• Permutation Inference
• builds a reference distribution
• reshuffle 0-1 values over locations
• recompute statistic (BB)
• compare observed statistic to reference distribution
• pseudo p-value = (M + 1)/(R + 1)
• M = values equal to or greater than statistic
• R = number of replications (e.g., 999)
Copyright © 2016 by Luc Anselin, All Rights Reserved
reference distribution, BB = 130
Copyright © 2016 by Luc Anselin, All Rights Reserved
reference distribution for spatially random valuesBB = 95
Copyright © 2016 by Luc Anselin, All Rights Reserved6
Moran’s I
Copyright © 2016 by Luc Anselin, All Rights Reserved
• Moran’s I
• the most commonly used of many spatial autocorrelation statistics
• I = [ Σi Σj wij zi.zj/S0 ]/[Σi zi2 / N]
• with zi = yi - mx : deviations from mean
• cross product statistic (zi.zj) similar to a correlation coefficient
• value depends on weights (wij)
Copyright © 2016 by Luc Anselin, All Rights Reserved
• Moran’s I examined more closely
• scaling factors in numerator and denominator
• in numerator: S0 = Σi Σj wij
• the number of non-zero elements in the weights matrix, or the number of neighbor pairs (x2)
• in denominator: N
• the total number of observations
Copyright © 2016 by Luc Anselin, All Rights Reserved
• Inference
• how to assess whether computed value of Moran’s I is significantly different from a value for a spatially random distribution
• compute analytically (assume normal distribution, etc.)
• computationally, compare value to a reference distribution obtained from a series of randomly permuted patterns
Copyright © 2016 by Luc Anselin, All Rights Reserved
• Approximate Inference
• assumes either normal distribution or randomization (each value equally likely to occur at any location)
• analytical derivation of moments (mean, variance) of the statistic under the null hypothesis (spatial randomness)
• z = (I - E[I]) / SD[I]
• approximate the resulting z-value by a standard normal distribution
Copyright © 2016 by Luc Anselin, All Rights Reserved
Cleveland 2015 q4 house sales prices (in $1,000)
Copyright © 2016 by Luc Anselin, All Rights Reserved
Normal Randomization
MI 0.282 0.282
E[MI] -0.0049 -0.0049
Var[MI] 0.00178 0.00158
z-value 6.81 7.22
p-value << 0.000001 << 0.000001
normal vs randomization inference for Moran’s Iqueen weights
Copyright © 2016 by Luc Anselin, All Rights Reserved
Queen K6 Distance
MI 0.282 0.343 0.335
E[MI] -0.0049 -0.0049 -0.0049
Var[MI] 0.00158 0.00124 0.00110
z-value 7.22 9.88 10.3
p-value << 0.000001 << 0.000001 << 0.000001
Moran’s Iinference under randomization, different spatial weights
Copyright © 2016 by Luc Anselin, All Rights Reserved
• Permutation Approach
• as such, even a high Moran’s I does not indicate significance
• need to construct a reference distribution
• randomly reshuffle observations and recompute Moran’s I each time
• compare observed value to reference distribution
Copyright © 2016 by Luc Anselin, All Rights Reserved
• Standardized z-value
• standardize by subtracting mean and dividing by standard deviation, computed from the reference distribution
• z = [Observed I - Mean(I)] / Standard Deviation(I)
• z-values are comparable across variables and across spatial weights
Copyright © 2016 by Luc Anselin, All Rights Reserved
999 permutations and reference distribution (queen weights)
MI = 0.282 Mean = -0.0045 s.d. = 0.0401z-value = 7.15
Copyright © 2016 by Luc Anselin, All Rights Reserved
Interpretation of Moran’s I
Copyright © 2016 by Luc Anselin, All Rights Reserved
• Sign of Moran’s I
• theoretical mean is - 1/(N - 1),
• essentially zero for large N
• positive and significant = clustering of like value
• NOT clustering of high or low
• could be either or a combination
• negative and significant = alternating values
• presence of spatial outliers
• spatial heterogeneity (checkerboard pattern)
Copyright © 2016 by Luc Anselin, All Rights Reserved
• Significance of Moran’s I
• compute pseudo p-value
• compare standard z-value to standard normal (approximate only)
• only if Moran’s I is significant is sign of coefficient meaningful
Copyright © 2016 by Luc Anselin, All Rights Reserved
• Comparing Moran’s I
• Moran’s I depends on spatial weights
• relative magnitude for same weights and different variables is meaningful
• but NOT for different spatial weights
• instead, use standardized z-value to compare
Copyright © 2016 by Luc Anselin, All Rights Reserved
Queen K6 Distance
MI 0.282 0.343 0.335
E[MI] -0.0049 -0.0049 -0.0049
Var[MI] 0.00158 0.00124 0.00110
z-value 7.22 9.88 10.3
p-value << 0.000001 << 0.000001 << 0.000001
Moran’s I, different spatial weightsMI is largest for K6, but z is largest for Distance
Copyright © 2016 by Luc Anselin, All Rights Reserved
• Clustering vs Clusters
• Moran’s I is a global statistic, i.e., a single value for the whole spatial pattern
• Moran’s I does NOT provide the location of clusters
• cluster detection requires a local statistic
Copyright © 2016 by Luc Anselin, All Rights Reserved
• True vs Apparent Contagion
• the indication of clustering does not provide an explanation for why the clustering occurs
• different processes can result in the same pattern
• true contagion: evidence of clustering due to spatial interaction (peer effects, mimicking, etc.)
• apparent contagion: evidence of clustering due to spatial heterogeneity (different spatial structures create local similarity)
Copyright © 2016 by Luc Anselin, All Rights Reserved
Geary’s c
Copyright © 2016 by Luc Anselin, All Rights Reserved
• Focus on Dissimilarity
• squared difference as measure of dissimilarity
• similar to notion of variogram (geostatistics)
• values between 0 and 2
Copyright © 2016 by Luc Anselin, All Rights Reserved
• Geary’s c Statistic
• c = (N-1) Σi Σj wij (xi - xj)2 / 2S0 Σ zi2
• alternativelyc = [ Σi Σj wij (xi - xj)2 / 2S0 ] / [ Σ zi2 / (N-1) ]
• with zi in deviations from the mean
• S0 = Σi Σj wij sum of all the weights
Copyright © 2016 by Luc Anselin, All Rights Reserved
• Interpretation
• positive spatial autocorrelationc < 1 or z < 0
• negative spatial autocorrelationc > 1 or z > 0
• opposite sign of Moran’s I
Copyright © 2016 by Luc Anselin, All Rights Reserved
• Inference
• same approach as for Moran’s I
• analytical
• approximate (normal, randomization)
• permutation
Copyright © 2016 by Luc Anselin, All Rights Reserved
Queen K6 Distance
c 0.716 0.496 0.544
E[c] 1.0 1.0 1.0
Var[c] 0.00381 0.00545 0.00285
z-value -4.61 -6.82 -8.55
p-value < 0.000004 << 0.000001 << 0.000001
geary’s cinference under randomization, different spatial weights
Copyright © 2016 by Luc Anselin, All Rights Reserved
999 permutations and reference distribution (queen weights)
c = 0.716 Mean = 0.998 s.d. = 0.062z-value = -4.58
Copyright © 2016 by Luc Anselin, All Rights Reserved6
Moran Scatter Plot
Copyright © 2016 by Luc Anselin, All Rights Reserved
• Recap: Moran’s I
• I = [ Σi Σj wij zi.zj/S0 ]/[Σi zi2 / N]
• with zi = yi - mx : deviations from mean
• for row-standardized weights S0 = N
• I = Σi Σj wij zi.zj / Σi zi2 = Σi zi (Σj wij.zj) / Σi zi2
• Moran’s I is slope in a regression of Σj wij.zj on zi
Copyright © 2016 by Luc Anselin, All Rights Reserved
• Moran Scatter Plot
• scatter plot of [ zi , Σj wij.zj ]
• the value at i on the x-axis, its spatial lag (weighted average of neighboring values) on the y-axis
• slope of linear fit is Moran’s I
• use local regression (Lowess) to identify possible structural breaks
Copyright © 2016 by Luc Anselin, All Rights Reserved
Moran scatter plot - Cleveland house prices - queen weights
outliers
Copyright © 2016 by Luc Anselin, All Rights Reserved
outliers (in red)
Copyright © 2016 by Luc Anselin, All Rights Reserved
Moran scatter plot without outliers
Copyright © 2016 by Luc Anselin, All Rights Reserved
• Categories of Local Spatial Autocorrelation
• four quadrants of the scatter plot
• upper right and lower left
• positive spatial autocorrelation
• clusters of like values
• locations are similar to their neighbors
• lower right and upper left
• negative spatial autocorrelation
• spatial outliers
• locations are different from their neighbors
Copyright © 2016 by Luc Anselin, All Rights Reserved
• Positive Spatial Autocorrelation
• all comparisons relative to the mean
• not absolute high or low
high-high low-low
Copyright © 2016 by Luc Anselin, All Rights Reserved
• Negative Spatial Autocorrelation
• spatial outliers
high-low low-high
Copyright © 2016 by Luc Anselin, All Rights Reserved
Moran scatter plot with Lowess smoother
Copyright © 2016 by Luc Anselin, All Rights Reserved6
Nonparametric Spatial Covariance Function
Copyright © 2016 by Luc Anselin, All Rights Reserved
• Alternative Perspective - Nonparametric
• non-parametric approach
• no model for covariance
• let the data suggest the functional form
• based on sample autocorrelation
• covariance function must be positive definite
Copyright © 2016 by Luc Anselin, All Rights Reserved
• Sample Autocorrelation
• computed for each pair i, j
• ρij = ρ(zi,zj) = zi*zj* / (1/n) Σh (zh - zm)2
• zi* = zi - zm deviations from mean
• in practice, easier to use standardized zi
• incidental parameter problem
• one parameter for each pair i-j
• n(n-1)/2 individual values of ρij
Copyright © 2016 by Luc Anselin, All Rights Reserved
• Non-Parametric Principle
• spatial autocorrelation as an unspecified function of distance
• ρij = g(dij)
• how to fit the function? use kernel estimator
Copyright © 2016 by Luc Anselin, All Rights Reserved
• Kernel Estimator
• Hall-Patil (1994)
• ρ(d) = Σi Σj K(dij/h)(zi.zj) / Σi Σj K(dij/h)
• K is the kernel function (many choices)
• h is the bandwidth
• results depends critically on h, less so on K
Copyright © 2016 by Luc Anselin, All Rights Reserved
• Kernel Regression
• zizj = g(dij)
• local regression
• depends on choice of kernel function and bandwidth
• values of the estimated g(dij) do not necessarily result in a valid (positive semi-definite) variance-covariance matrix
Copyright © 2016 by Luc Anselin, All Rights Reserved
correlogram - full distance range
Copyright © 2016 by Luc Anselin, All Rights Reserved
correlogram - 20,000 ft max distance
Copyright © 2016 by Luc Anselin, All Rights Reserved
• Interpretation
• range of spatial autocorrelation
• alternative to specifying spatial weights
• sensitive to kernel fit
• may violate Tobler’s law