References
Evaluating sensitivity of parametersof interest to measurement invariance
using the EPC-interest
Daniel Oberski
Department of methodology and statistics, Tilburg University
WorkingGroupStructuralEquationModeling26-27.02.2015, FU Berlin
Measurement invariance using the EPC-interest Daniel Oberski
References
Conclusion
• This talk discusses the ``EPC-interest''• EPC-interest is like SEM's expected parameter change
(``EPC-self'') but instead of measuring the change in therestricted parameter, it measures the change in theparameter of interest;
• Introduced for measurement invariance evaluation incontinuous data SEM by Oberski (2014) and forcategorical data by Oberski et al. (frth)
• Implemented in Latent Gold ≥5.0, experimental version oflavaan, working on main branch.
Measurement invariance using the EPC-interest Daniel Oberski
References
The problem of measurement invariance
Measurement invariance using the EPC-interest Daniel Oberski
References
p(y) =∑ξ
p(ξ|x)J∏j=1
p(yj|ξ)
Measurement invariance using the EPC-interest Daniel Oberski
References
p(y) =∑ξ
p(ξ|x)J∏j=1
p(yj|ξ, x)
Measurement invariance using the EPC-interest Daniel Oberski
References
Measurement invariance: the problem
Problem of measurement invariance: we want to know γ,but δ = 0 might bias this parameterofinterest.
Measurement invariance using the EPC-interest Daniel Oberski
References
Preceding solutions to the problem
..1 Selecting one indicator as a reference indicator (``anchoritem'') apriori;
..2 Imposing a strong prior on differential functioning(Muthén&Asparouhov2012)
..3 Test null hypothesis of full or partial invariance.
..4 Sensitivity analysis: allow partial violations when theymatter.
Measurement invariance using the EPC-interest Daniel Oberski
References
Preceding solutions to the problem
..1 Selecting one indicator as a reference indicator (``anchoritem'') apriori;
+ Don't need further restrictions;- No way of testing reference indicator.
E.g.: Setting loadings to 1 in each group, ``alignment method''.
..2 Imposing a strong prior on differential functioning(Muthén&Asparouhov2012)
..3 Test null hypothesis of full or partial invariance.
..4 Sensitivity analysis: allow partial violations when theymatter.
Measurement invariance using the EPC-interest Daniel Oberski
References
Preceding solutions to the problem
..1 Selecting one indicator as a reference indicator (``anchoritem'') apriori;
..2 Imposing a strong prior on differential functioning(Muthén&Asparouhov2012)
+ Attractive data-driven solution when prior is neither tooinformative nor too weak;
- More research needed to figure out when prior is neithertoo informative nor too weak.
..3 Test null hypothesis of full or partial invariance.
..4 Sensitivity analysis: allow partial violations when theymatter.
Measurement invariance using the EPC-interest Daniel Oberski
References
Preceding solutions to the problem..1 Selecting one indicator as a reference indicator (``anchor
item'') apriori;..2 Imposing a strong prior on differential functioning(Muthén&Asparouhov2012)
..3 Test null hypothesis of full or partial invariance.+ When high-powered test of full measurement invariance is
not rejected, may be safe to simply continue without theneed for further modeling;
- Rarely happens in practice.+ Partial invariance looks at size and significance of δ, but
not all big δ's are important nor are small ones necessarilyunimportant. So does not guarantee that he parameter ofinterest in free of measurement differences (Oberski,2014)
..4 Sensitivity analysis: allow partial violations when theymatter.
Measurement invariance using the EPC-interest Daniel Oberski
References
Preceding solutions to the problem
..1 Selecting one indicator as a reference indicator (``anchoritem'') apriori;
..2 Imposing a strong prior on differential functioning(Muthén&Asparouhov2012)
..3 Test null hypothesis of full or partial invariance.
..4 Sensitivity analysis: allow partial violations when theymatter.
Measurement invariance using the EPC-interest Daniel Oberski
References
Measurement invariance: the problem and a solution
Problem of measurement invariance: we want to know γ,but δ = 0 might bias this parameterofinterest.
Solution: Use EPC-interest: the expected change in γ whenfreeing δ.
• If EPC-interest is big (e.g. can change sign), incorporate δ;• if EPC-interest is small, ignore it.
Measurement invariance using the EPC-interest Daniel Oberski
References
EPC-interest
Measurement invariance using the EPC-interest Daniel Oberski
References
EPC-interest, also for categorical data
EPC-interest = P(∂θ
∂δ′
)(δ − δ
)= −PH−1
θθ Hθδ EPC-self= γa − γ +O(ψ′ψ),
where P selects the parameters of interest γ from theparameter vector θ, H is a Hessian, and O(ψ′ψ) is anapproximation term depending on the overall amount ofmisspecification (parameter differences).
Measurement invariance using the EPC-interest Daniel Oberski
References
Simulation: how good is the approximation?Setup:
P(Yj = 1|x) = [1 + exp(−x)]−1,
with j ∈ {2, 3, 4}, and structural model
x = γz+ ϵ
with γ = 1 and ϵ ∼ N(0, 1). We then introduced a violation ofmeasurement invariance for the first indicator,
P(Y1 = 1|x) = [1 + exp(−x− δz)]−1.
Nine conditions varied sample size, n ∈ {250, 500, 1000}, andthe size of the invariance violation: δ = 0 (no violation), 0.5(moderate), or 1 (extreme). Data were generated using R 3.1.2and analyzed using Latent GOLD 5.0.0.14161.
Measurement invariance using the EPC-interest Daniel Oberski
References
n 250 500 1000True δ 0 0.5 1 0 0.5 1 0 0.5 1Est. γ 1.010 1.151 1.353 0.980 1.152 1.330 1.013 1.163 1.327Bias γ -0.010 -0.151 -0.353 0.020 -0.152 -0.330 -0.013 -0.163 -0.327EPC-int. 0.003 -0.166 -0.494 -0.001 -0.180 -0.486 0.004 -0.182 -0.448
Table : Simulation study of EPC-interest. Shown is the average pointestimate for the γ parameter of interest under full measurementinvariance (``Est''), its difference from the true value γ = 1 (``Bias''),and the average EPC-interest.
Measurement invariance using the EPC-interest Daniel Oberski
References
Example with categorical dataSee http://daob.nl/publications/
Measurement invariance using the EPC-interest Daniel Oberski
References
Figure : Graphical representation of the multilevel latent classregression model for (post)materialism measured by three partialranking tasks. Observed variables are shown in rectangles whileunobserved (``latent'') variables are shown in ellipses.
Measurement invariance using the EPC-interest Daniel Oberski
References
Multilevel latent class model w/ covariates forrankings
L(θ) = P(A1,A2,B1,B2,C1,C2|Z1,Z2) =
C∏c=1
∑G
P(Gc)nc∏i=1
∑X
P(Xic|Z1ic,Z2ic,Gc)×
P(A1ic,A2ic|Xic)P(B1ic,B2ic|Xic)P(C1ic,C2ic|Xic),
Goal: estimate γ (especially its sign).Possibleproblem: Violations of scalar and metricmeasurement invariance (DIF), parameterized respectivelyas τ∗ and λ∗.Solution: See if these matter for the sign of γ.
Measurement invariance using the EPC-interest Daniel Oberski
References
Table : Full invariance multilevel latent class model: EPC-interestvalues.
λ∗jkxg
Estimates EPC-interestEst. s.e. Task 1 Task 2 Task 3
Class 1 GDP -0.035 (0.007) 0.073 0.252 0.005Class 2 GDP -0.198 (0.012) -0.163 -0.058 0.002
Class 1 Women 0.013 (0.001) -0.003 0.029 0.002Class 2 Women -0.037 (0.001) -0.006 -0.013 0.002
Free ``loadings'' for task 1 and task 2.
Measurement invariance using the EPC-interest Daniel Oberski
References
Table : Partially invariant multilevel latent class model: EPC-interestvalues.
λ∗jkxg
Estimates EPC-interestEst. s.e. Task 1 Task 2 Task 3
Class 1 GDP -0.127 (0.008) 0.097Class 2 GDP 0.057 (0.011) 0.161
Class 1 Women 0.008 (0.001) 0.001Class 2 Women 0.020 (0.001) 0.007
Measurement invariance using the EPC-interest Daniel Oberski
References
WhathasbeengainedbyusingEPC-interest:
I am fairly confident here that there truly is "approximatemeasurement invariance", in the sense that any violations ofmeasurement invariance do not bias the primary conclusions.
I think attaining this goal is the main purpose of model fitevaluation.
Measurement invariance using the EPC-interest Daniel Oberski
References
Conclusion
Measurement invariance using the EPC-interest Daniel Oberski
References
Whatistheproblem?• We do latent variable modeling with a goal in mind.• But the latent variable model might be misspecified.• The appropriate question: "will that affect my goal?"• The actual question: "do the data fit the model in the
population" (LR) or "are the model and the data far apartrelative to model complexity" (RMSEA etc.)
Whatisthesolution?
Evaluatedirectlywhateffectpossiblemisspecificationshaveonthegoaloftheanalysis.
Measurement invariance using the EPC-interest Daniel Oberski
References
References
Oberski, D. (2014). Evaluating sensitivity of parameters of interest tomeasurement invariance in latent variable models. PoliticalAnalysis, 22(1):45--60.
Oberski, D. and Vermunt, J. (2013). A model-based approach togoodness-of-fit evaluation in item response theory. Measurement:InterdisciplinaryResearch&Perspectives, 11:117--122.
Oberski, D., Vermunt, J., and Moors, G. (frth). Evaluatingmeasurement invariance in categorical data latent variable modelswith the EPC-interest.
Vermunt, J. K. and Magidson, J. (2013). TechnicalguideforLatentGOLD 5.0: Basicandadvanced. Statistical Innovations Inc.,Belmont, MA.
Measurement invariance using the EPC-interest Daniel Oberski
SEM regression coefficient example
Two problems with invariance testing
Misspecified invariance model fit``Good'' fit ``Bad'' fit
ConclusionsUnaffectedby misspec-ification
(1) √ (2) Overparameteriza-tion or unnecessarilyd i s c a r d e d i t e m ,group, or scale.
Affected bymisspecifi-cation
(3) Non-invariance in-validates conclusions.
(4) √
Measurement invariance using the EPC-interest Daniel Oberski
SEM regression coefficient example
European Sociological Review 2008, 24(5), 583--599Measurement invariance using the EPC-interest Daniel Oberski
SEM regression coefficient example
Conservation Self−transcendence
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
SwedenDanmark
AustriaSwitzerlandNetherlands
GermanyIrelandSpain
NorwayHungaryFinland
PortugalFrance
BelgiumSlovenia
United KingdomGreece
Czech RepublicPoland
SwedenDanmark
AustriaSwitzerlandNetherlands
GermanyIrelandSpain
NorwayHungaryFinland
PortugalFrance
BelgiumSlovenia
United KingdomGreece
Czech RepublicPoland
ALLO
WN
OC
ON
D
−1.0 −0.5 0.0 0.5 1.0 −1.0 −0.5 0.0 0.5 1.0Regression coefficient
Measurement invariance using the EPC-interest Daniel Oberski
SEM regression coefficient example
EPC-interest statistics of at least 0.1 in absolute value withrespect to the latent variable regression coefficients.
Metric invariance (loading) restriction``Conditions → Work skills'' in...
Slovenia France Hungary IrelandEPC-interest w.r.t.:Conditions →
Self-transcendence -0.073 -0.092 -0.067 0.073Conservation 0.144 0.139 0.123 -0.113
SEPC-self 0.610 0.692 0.759 -0.514
Measurement invariance using the EPC-interest Daniel Oberski
SEM regression coefficient example
What has been gained by using EPC-interest
• Full metric invariance model: "close fit";• EPC-interest still detects threats to cross-country
comparisons of regression coefficients;
• MI and EPC-self do not detect these particularmisspecifications;
• MI and EPC-self detect other misspecifications;• Looking at EPC-interest reveals that these do not affect
the cross-country comparisons of regression coefficients.
Measurement invariance using the EPC-interest Daniel Oberski
SEM regression coefficient example
What has been gained by using EPC-interest
• Full metric invariance model: "close fit";• EPC-interest still detects threats to cross-country
comparisons of regression coefficients;• MI and EPC-self do not detect these particular
misspecifications;
• MI and EPC-self detect other misspecifications;• Looking at EPC-interest reveals that these do not affect
the cross-country comparisons of regression coefficients.
Measurement invariance using the EPC-interest Daniel Oberski
SEM regression coefficient example
What has been gained by using EPC-interest
• Full metric invariance model: "close fit";• EPC-interest still detects threats to cross-country
comparisons of regression coefficients;• MI and EPC-self do not detect these particular
misspecifications;• MI and EPC-self detect other misspecifications;
• Looking at EPC-interest reveals that these do not affectthe cross-country comparisons of regression coefficients.
Measurement invariance using the EPC-interest Daniel Oberski
SEM regression coefficient example
What has been gained by using EPC-interest
• Full metric invariance model: "close fit";• EPC-interest still detects threats to cross-country
comparisons of regression coefficients;• MI and EPC-self do not detect these particular
misspecifications;• MI and EPC-self detect other misspecifications;• Looking at EPC-interest reveals that these do not affect
the cross-country comparisons of regression coefficients.
Measurement invariance using the EPC-interest Daniel Oberski
SEM regression coefficient example
What has been gained by using EPC-interest
• Full metric invariance model: "close fit";• EPC-interest still detects threats to cross-country
comparisons of regression coefficients;• MI and EPC-self do not detect these particular
misspecifications;• MI and EPC-self detect other misspecifications;• Looking at EPC-interest reveals that these do not affect
the cross-country comparisons of regression coefficients.
Measurement invariance using the EPC-interest Daniel Oberski
ExampleGoal: Estimate gender differences in "valuing Stimulation":
(1) Very much like me; (2) Like me; (3) Somewhat like me; (4) Alittle like me; (5) Not like me; (6) Not like me at all.
impdiff (S)he looks for adventures and likes to take risks.(S)he wants to have an exciting life.
impadv (S)he likes surprises and is always looking for newthings to do. He thinks it is important to do lots ofdifferent things in life.
Tool: Structural Equation Model for European Social Surveydata (n = 18519 men and 16740 women).(OriginalstudybySchwarzetal. 2005)
Measurement invariance using the EPC-interest Daniel Oberski
Assume: Butreally(?):
What difference does it make for the goal: true genderdifferences in values? (re-analysisofdatabyOberski2014)
●
●●
●
●
●
●
●
Men value moreWomen value more
−0.2
0.0
0.2
ACPO ST SD HE COTR SE UN BE"Human value" factor
Late
nt m
ean
diffe
renc
e es
timat
e ±
2 s.
e.
Model
● Scalar invariance
Free intercept 'Adventure'
Measurement invariance using the EPC-interest Daniel Oberski
PROBLEM
The original authors found that the conditional independencemodel fit the data "approximately" (p. 1013)...
"Chi-squaredeterioratedsignificantly, ∆χ2(19) = 3313,p < .001, butCFI didnotchange. Changeinchi-squareishighlysensitivewithlargesamplesizesandcomplexmodels. Theotherindicessuggestedthatscalarinvariancemightbeaccepted(CFI =.88, RMSEA =.04, CI =.039.040,PCLOSE =1.0).''
... but unfortunately this "acceptable" misspecification couldreversetheirconclusions!
Measurement invariance using the EPC-interest Daniel Oberski
PROBLEM
The original authors found that the conditional independencemodel fit the data "approximately" (p. 1013)...
"Chi-squaredeterioratedsignificantly, ∆χ2(19) = 3313,p < .001, butCFI didnotchange. Changeinchi-squareishighlysensitivewithlargesamplesizesandcomplexmodels. Theotherindicessuggestedthatscalarinvariancemightbeaccepted(CFI =.88, RMSEA =.04, CI =.039.040,PCLOSE =1.0).''
... but unfortunately this "acceptable" misspecification couldreversetheirconclusions!
Measurement invariance using the EPC-interest Daniel Oberski
EPC-interest applied to Stimulation example
• After fitting the full scalar invariance model,• Effect size estimate of sex difference in Stimulation is
+0.214 (s.e. 0.0139).
• But EPC-interest of equal "Adventure" item intercept is-0.243.
• So EPC-interest suggests conclusioncanbereversedby freeing a misspecified scalar invariance restriction
• Actualchange when freeing this intercept is very close toEPC-interest: -0.235.
Measurement invariance using the EPC-interest Daniel Oberski
EPC-interest applied to Stimulation example
• After fitting the full scalar invariance model,• Effect size estimate of sex difference in Stimulation is
+0.214 (s.e. 0.0139).• But EPC-interest of equal "Adventure" item intercept is-0.243.
• So EPC-interest suggests conclusioncanbereversedby freeing a misspecified scalar invariance restriction
• Actualchange when freeing this intercept is very close toEPC-interest: -0.235.
Measurement invariance using the EPC-interest Daniel Oberski
EPC-interest applied to Stimulation example
• After fitting the full scalar invariance model,• Effect size estimate of sex difference in Stimulation is
+0.214 (s.e. 0.0139).• But EPC-interest of equal "Adventure" item intercept is-0.243.
• So EPC-interest suggests conclusioncanbereversedby freeing a misspecified scalar invariance restriction
• Actualchange when freeing this intercept is very close toEPC-interest: -0.235.
Measurement invariance using the EPC-interest Daniel Oberski
EPC-interest applied to Stimulation example
• After fitting the full scalar invariance model,• Effect size estimate of sex difference in Stimulation is
+0.214 (s.e. 0.0139).• But EPC-interest of equal "Adventure" item intercept is-0.243.
• So EPC-interest suggests conclusioncanbereversedby freeing a misspecified scalar invariance restriction
• Actualchange when freeing this intercept is very close toEPC-interest: -0.235.
Measurement invariance using the EPC-interest Daniel Oberski
EPC-interest applied to Stimulation example
• After fitting the full scalar invariance model,• Effect size estimate of sex difference in Stimulation is
+0.214 (s.e. 0.0139).• But EPC-interest of equal "Adventure" item intercept is-0.243.
• So EPC-interest suggests conclusioncanbereversedby freeing a misspecified scalar invariance restriction
• Actualchange when freeing this intercept is very close toEPC-interest: -0.235.
Measurement invariance using the EPC-interest Daniel Oberski