Strictly Positive Estimates of Variance Componentsfor Measurement Systems Analysis Models
Laura Lancaster and Chris Gotwalt
JMP Research & DevelopmentSAS Institute
Discovery Summit 2015
Laura Lancaster (SAS Institute) Discovery Summit 2015 1 / 39
Outline
1 Measurement Systems Analysis
2 Bayesian Estimation Method
3 Simulation Study
4 Two Factors Crossed Study
5 Two Factors Nested Study
6 Three Factors Staggered Nested Study
7 Conclusions
Laura Lancaster (SAS Institute) Discovery Summit 2015 2 / 39
Measurement Systems Analysis (MSA)
Designed experiments that help determine how muchmeasurement variation is contributing to the overall processvariation.
Model for an observed measurement
X = P + EX = P + EX = P + E
I XXX = Observed Product MeasurementI PPP = Product ValueI EEE = Measurement ErrorI PPP and E are assumed independent with P ∼ N(µp, σ
2p)P ∼ N(µp, σ2p)P ∼ N(µp, σ2p) and
E ∼ N(µe, σ2e)E ∼ N(µe, σ2e)E ∼ N(µe, σ2e)
I We will assume that µe = 0µe = 0µe = 0.
Observed Measurement Variance
σ2x = σ2
p + σ2eσ2
x = σ2p + σ2
eσ2x = σ2
p + σ2e
Laura Lancaster (SAS Institute) Discovery Summit 2015 3 / 39
Measurement Systems Analysis (MSA)
Designed experiments that help determine how muchmeasurement variation is contributing to the overall processvariation.Model for an observed measurement
X = P + EX = P + EX = P + E
I XXX = Observed Product MeasurementI PPP = Product ValueI EEE = Measurement ErrorI PPP and E are assumed independent with P ∼ N(µp, σ
2p)P ∼ N(µp, σ2p)P ∼ N(µp, σ2p) and
E ∼ N(µe, σ2e)E ∼ N(µe, σ2e)E ∼ N(µe, σ2e)
I We will assume that µe = 0µe = 0µe = 0.
Observed Measurement Variance
σ2x = σ2
p + σ2eσ2
x = σ2p + σ2
eσ2x = σ2
p + σ2e
Laura Lancaster (SAS Institute) Discovery Summit 2015 3 / 39
Measurement Systems Analysis (MSA)
Designed experiments that help determine how muchmeasurement variation is contributing to the overall processvariation.Model for an observed measurement
X = P + EX = P + EX = P + E
I XXX = Observed Product MeasurementI PPP = Product ValueI EEE = Measurement ErrorI PPP and E are assumed independent with P ∼ N(µp, σ
2p)P ∼ N(µp, σ2p)P ∼ N(µp, σ2p) and
E ∼ N(µe, σ2e)E ∼ N(µe, σ2e)E ∼ N(µe, σ2e)
I We will assume that µe = 0µe = 0µe = 0.
Observed Measurement Variance
σ2x = σ2
p + σ2eσ2
x = σ2p + σ2
eσ2x = σ2
p + σ2e
Laura Lancaster (SAS Institute) Discovery Summit 2015 3 / 39
MSA Models
Use random effects models to estimate the sources ofvariation in the measurement process.
Variance components are typically estimated using one of threemethods:
I Average and Range MethodI Expected Means Squares (EMS)I Restricted Maximum Likelihood (REML)
These methods can produce zeroed variance components!
Laura Lancaster (SAS Institute) Discovery Summit 2015 4 / 39
MSA Models
Use random effects models to estimate the sources ofvariation in the measurement process.Variance components are typically estimated using one of threemethods:
I Average and Range MethodI Expected Means Squares (EMS)I Restricted Maximum Likelihood (REML)
These methods can produce zeroed variance components!
Laura Lancaster (SAS Institute) Discovery Summit 2015 4 / 39
MSA Models
Use random effects models to estimate the sources ofvariation in the measurement process.Variance components are typically estimated using one of threemethods:
I Average and Range MethodI Expected Means Squares (EMS)I Restricted Maximum Likelihood (REML)
These methods can produce zeroed variance components!
Laura Lancaster (SAS Institute) Discovery Summit 2015 4 / 39
Motivation
Wanted to find an estimation method that produces strictlypositive variance components and has good properties for MSAmodels.
There are several non-informative prior methods that producestrictly positive variance components for random effects models.
I A flat prior on (0,∞) - Not goodI Jeffrey’s prior - Not goodI Portnoy and Sahai’s modified Jeffrey’s Prior - This prior had
posterior means with good properties!
We generalized Portnoy and Sahai’s approach and implemented itin JMP’s Variability platform.Default JMP behavior is to use REML estimates if no variancecomponents have been zeroed and use the Bayesian estimatesotherwise. We will refer to this as a Hybrid method.
Laura Lancaster (SAS Institute) Discovery Summit 2015 5 / 39
Motivation
Wanted to find an estimation method that produces strictlypositive variance components and has good properties for MSAmodels.
There are several non-informative prior methods that producestrictly positive variance components for random effects models.
I A flat prior on (0,∞) - Not goodI Jeffrey’s prior - Not goodI Portnoy and Sahai’s modified Jeffrey’s Prior - This prior had
posterior means with good properties!
We generalized Portnoy and Sahai’s approach and implemented itin JMP’s Variability platform.Default JMP behavior is to use REML estimates if no variancecomponents have been zeroed and use the Bayesian estimatesotherwise. We will refer to this as a Hybrid method.
Laura Lancaster (SAS Institute) Discovery Summit 2015 5 / 39
Motivation
Wanted to find an estimation method that produces strictlypositive variance components and has good properties for MSAmodels.
There are several non-informative prior methods that producestrictly positive variance components for random effects models.
I A flat prior on (0,∞)
- Not goodI Jeffrey’s prior - Not goodI Portnoy and Sahai’s modified Jeffrey’s Prior - This prior had
posterior means with good properties!
We generalized Portnoy and Sahai’s approach and implemented itin JMP’s Variability platform.Default JMP behavior is to use REML estimates if no variancecomponents have been zeroed and use the Bayesian estimatesotherwise. We will refer to this as a Hybrid method.
Laura Lancaster (SAS Institute) Discovery Summit 2015 5 / 39
Motivation
Wanted to find an estimation method that produces strictlypositive variance components and has good properties for MSAmodels.
There are several non-informative prior methods that producestrictly positive variance components for random effects models.
I A flat prior on (0,∞) - Not good
I Jeffrey’s prior - Not goodI Portnoy and Sahai’s modified Jeffrey’s Prior - This prior had
posterior means with good properties!
We generalized Portnoy and Sahai’s approach and implemented itin JMP’s Variability platform.Default JMP behavior is to use REML estimates if no variancecomponents have been zeroed and use the Bayesian estimatesotherwise. We will refer to this as a Hybrid method.
Laura Lancaster (SAS Institute) Discovery Summit 2015 5 / 39
Motivation
Wanted to find an estimation method that produces strictlypositive variance components and has good properties for MSAmodels.
There are several non-informative prior methods that producestrictly positive variance components for random effects models.
I A flat prior on (0,∞) - Not goodI Jeffrey’s prior
- Not goodI Portnoy and Sahai’s modified Jeffrey’s Prior - This prior had
posterior means with good properties!
We generalized Portnoy and Sahai’s approach and implemented itin JMP’s Variability platform.Default JMP behavior is to use REML estimates if no variancecomponents have been zeroed and use the Bayesian estimatesotherwise. We will refer to this as a Hybrid method.
Laura Lancaster (SAS Institute) Discovery Summit 2015 5 / 39
Motivation
Wanted to find an estimation method that produces strictlypositive variance components and has good properties for MSAmodels.
There are several non-informative prior methods that producestrictly positive variance components for random effects models.
I A flat prior on (0,∞) - Not goodI Jeffrey’s prior - Not good
I Portnoy and Sahai’s modified Jeffrey’s Prior - This prior hadposterior means with good properties!
We generalized Portnoy and Sahai’s approach and implemented itin JMP’s Variability platform.Default JMP behavior is to use REML estimates if no variancecomponents have been zeroed and use the Bayesian estimatesotherwise. We will refer to this as a Hybrid method.
Laura Lancaster (SAS Institute) Discovery Summit 2015 5 / 39
Motivation
Wanted to find an estimation method that produces strictlypositive variance components and has good properties for MSAmodels.
There are several non-informative prior methods that producestrictly positive variance components for random effects models.
I A flat prior on (0,∞) - Not goodI Jeffrey’s prior - Not goodI Portnoy and Sahai’s modified Jeffrey’s Prior
- This prior hadposterior means with good properties!
We generalized Portnoy and Sahai’s approach and implemented itin JMP’s Variability platform.Default JMP behavior is to use REML estimates if no variancecomponents have been zeroed and use the Bayesian estimatesotherwise. We will refer to this as a Hybrid method.
Laura Lancaster (SAS Institute) Discovery Summit 2015 5 / 39
Motivation
Wanted to find an estimation method that produces strictlypositive variance components and has good properties for MSAmodels.
There are several non-informative prior methods that producestrictly positive variance components for random effects models.
I A flat prior on (0,∞) - Not goodI Jeffrey’s prior - Not goodI Portnoy and Sahai’s modified Jeffrey’s Prior - This prior had
posterior means with good properties!
We generalized Portnoy and Sahai’s approach and implemented itin JMP’s Variability platform.Default JMP behavior is to use REML estimates if no variancecomponents have been zeroed and use the Bayesian estimatesotherwise. We will refer to this as a Hybrid method.
Laura Lancaster (SAS Institute) Discovery Summit 2015 5 / 39
Motivation
Wanted to find an estimation method that produces strictlypositive variance components and has good properties for MSAmodels.
There are several non-informative prior methods that producestrictly positive variance components for random effects models.
I A flat prior on (0,∞) - Not goodI Jeffrey’s prior - Not goodI Portnoy and Sahai’s modified Jeffrey’s Prior - This prior had
posterior means with good properties!
We generalized Portnoy and Sahai’s approach and implemented itin JMP’s Variability platform.
Default JMP behavior is to use REML estimates if no variancecomponents have been zeroed and use the Bayesian estimatesotherwise. We will refer to this as a Hybrid method.
Laura Lancaster (SAS Institute) Discovery Summit 2015 5 / 39
Motivation
Wanted to find an estimation method that produces strictlypositive variance components and has good properties for MSAmodels.
There are several non-informative prior methods that producestrictly positive variance components for random effects models.
I A flat prior on (0,∞) - Not goodI Jeffrey’s prior - Not goodI Portnoy and Sahai’s modified Jeffrey’s Prior - This prior had
posterior means with good properties!
We generalized Portnoy and Sahai’s approach and implemented itin JMP’s Variability platform.Default JMP behavior is to use REML estimates if no variancecomponents have been zeroed and use the Bayesian estimatesotherwise. We will refer to this as a Hybrid method.
Laura Lancaster (SAS Institute) Discovery Summit 2015 5 / 39
Research Questions
1 How do the Bayesian estimates compare to the REMLestimates?
I Bias?I Variability?I Classification?
• Don Wheeler’s Evaluating the Measurement Process (EMP) method• Automotive Industry Action Group’s (AIAG) Gauge R&R method
2 How does our Hybrid method perform?
Laura Lancaster (SAS Institute) Discovery Summit 2015 6 / 39
Research Questions
1 How do the Bayesian estimates compare to the REMLestimates?
I Bias?
I Variability?I Classification?
• Don Wheeler’s Evaluating the Measurement Process (EMP) method• Automotive Industry Action Group’s (AIAG) Gauge R&R method
2 How does our Hybrid method perform?
Laura Lancaster (SAS Institute) Discovery Summit 2015 6 / 39
Research Questions
1 How do the Bayesian estimates compare to the REMLestimates?
I Bias?I Variability?
I Classification?• Don Wheeler’s Evaluating the Measurement Process (EMP) method• Automotive Industry Action Group’s (AIAG) Gauge R&R method
2 How does our Hybrid method perform?
Laura Lancaster (SAS Institute) Discovery Summit 2015 6 / 39
Research Questions
1 How do the Bayesian estimates compare to the REMLestimates?
I Bias?I Variability?I Classification?
• Don Wheeler’s Evaluating the Measurement Process (EMP) method• Automotive Industry Action Group’s (AIAG) Gauge R&R method
2 How does our Hybrid method perform?
Laura Lancaster (SAS Institute) Discovery Summit 2015 6 / 39
Research Questions
1 How do the Bayesian estimates compare to the REMLestimates?
I Bias?I Variability?I Classification?
• Don Wheeler’s Evaluating the Measurement Process (EMP) method• Automotive Industry Action Group’s (AIAG) Gauge R&R method
2 How does our Hybrid method perform?
Laura Lancaster (SAS Institute) Discovery Summit 2015 6 / 39
Wheeler’s EMP Classification System
Intraclass Correlation Coefficient (ICC) - ρρρ - ratio of productvariance to total variance
ρ =σ2
p
σ2p + σ2
e=σ2
p
σ2x
ρ =σ2
p
σ2p + σ2
e=σ2
p
σ2x
ρ =σ2
p
σ2p + σ2
e=σ2
p
σ2x
Wheeler’s EMP Classifications:
Classification ρ̂̂ρ̂ρ Prob 3σp3σp3σp Shift - Warning Rule 1First Class 0.80− 1.00 0.99− 1.00Second Class 0.50− 0.80 0.88− 0.99Third Class 0.20− 0.50 0.40− 0.88Fourth Class 0.00− 0.20 0.03− 0.40
Laura Lancaster (SAS Institute) Discovery Summit 2015 7 / 39
Wheeler’s EMP Classification System
Intraclass Correlation Coefficient (ICC) - ρρρ - ratio of productvariance to total variance
ρ =σ2
p
σ2p + σ2
e=σ2
p
σ2x
ρ =σ2
p
σ2p + σ2
e=σ2
p
σ2x
ρ =σ2
p
σ2p + σ2
e=σ2
p
σ2x
Wheeler’s EMP Classifications:
Classification ρ̂̂ρ̂ρ Prob 3σp3σp3σp Shift - Warning Rule 1First Class 0.80− 1.00 0.99− 1.00Second Class 0.50− 0.80 0.88− 0.99Third Class 0.20− 0.50 0.40− 0.88Fourth Class 0.00− 0.20 0.03− 0.40
Laura Lancaster (SAS Institute) Discovery Summit 2015 7 / 39
AIAG’s Classification System
AIAG uses Percent Gauge R&R to classify the health of ameasurement system.
%GRR = 100
√σ̂2
e
σ̂2p + σ̂2
e= 100
σ̂e
σ̂x= 100
√1− ρ̂%GRR = 100
√σ̂2
e
σ̂2p + σ̂2
e= 100
σ̂e
σ̂x= 100
√1− ρ̂%GRR = 100
√σ̂2
e
σ̂2p + σ̂2
e= 100
σ̂e
σ̂x= 100
√1− ρ̂
AIAG Classifications:
Classification %GRR%GRR%GRR ρ̂̂ρ̂ρ
Acceptable 0%− 10% 0.99− 1.00Marginal 10%− 30% 0.91− 0.99Unacceptable 30%− 100% 0.00− 0.91
Laura Lancaster (SAS Institute) Discovery Summit 2015 8 / 39
AIAG’s Classification System
AIAG uses Percent Gauge R&R to classify the health of ameasurement system.
%GRR = 100
√σ̂2
e
σ̂2p + σ̂2
e= 100
σ̂e
σ̂x= 100
√1− ρ̂%GRR = 100
√σ̂2
e
σ̂2p + σ̂2
e= 100
σ̂e
σ̂x= 100
√1− ρ̂%GRR = 100
√σ̂2
e
σ̂2p + σ̂2
e= 100
σ̂e
σ̂x= 100
√1− ρ̂
AIAG Classifications:
Classification %GRR%GRR%GRR ρ̂̂ρ̂ρ
Acceptable 0%− 10% 0.99− 1.00Marginal 10%− 30% 0.91− 0.99Unacceptable 30%− 100% 0.00− 0.91
Laura Lancaster (SAS Institute) Discovery Summit 2015 8 / 39
Portnoy-Sahai Estimator
Modified Jeffrey’s Prior:
πPS(σ2i ) = |F(σ2)|
12 /τ4
i
I σ2i - the i th variance component.
I τ2i - expected mean square associated with the σ2
i .I F(σ2) - Fisher information matrix.
The Portnoy-Sahai estimator, σ̂2i , is the posterior mean of σ2
i usingthe modified Jeffrey’s prior.We use Gauss-Laguerre quadrature to perform the numericalintegration for these estimators.
Laura Lancaster (SAS Institute) Discovery Summit 2015 9 / 39
Portnoy-Sahai Estimator
Modified Jeffrey’s Prior:
πPS(σ2i ) = |F(σ2)|
12 /τ4
i
I σ2i - the i th variance component.
I τ2i - expected mean square associated with the σ2
i .I F(σ2) - Fisher information matrix.
The Portnoy-Sahai estimator, σ̂2i , is the posterior mean of σ2
i usingthe modified Jeffrey’s prior.
We use Gauss-Laguerre quadrature to perform the numericalintegration for these estimators.
Laura Lancaster (SAS Institute) Discovery Summit 2015 9 / 39
Portnoy-Sahai Estimator
Modified Jeffrey’s Prior:
πPS(σ2i ) = |F(σ2)|
12 /τ4
i
I σ2i - the i th variance component.
I τ2i - expected mean square associated with the σ2
i .I F(σ2) - Fisher information matrix.
The Portnoy-Sahai estimator, σ̂2i , is the posterior mean of σ2
i usingthe modified Jeffrey’s prior.We use Gauss-Laguerre quadrature to perform the numericalintegration for these estimators.
Laura Lancaster (SAS Institute) Discovery Summit 2015 9 / 39
Simulation Study Design
Three Typical MSA ModelsI Two Factors Crossed (balanced)I Two Factors Nested (balanced)I Three Factors Staggered Nested Design (unbalanced)
Range of bad to good measurement systems (using ICC as themetric)
I ICC values in middle of Wheeler’s EMP classifications:0.1, 0.35, 0.65, 0.9
I ICC values in middle of AIAG’s top 2 classifications:0.96 and 0.995
Part variance values:1, 5, and 25
Laura Lancaster (SAS Institute) Discovery Summit 2015 10 / 39
Simulation Study Design
Three Typical MSA ModelsI Two Factors Crossed (balanced)I Two Factors Nested (balanced)I Three Factors Staggered Nested Design (unbalanced)
Range of bad to good measurement systems (using ICC as themetric)
I ICC values in middle of Wheeler’s EMP classifications:0.1, 0.35, 0.65, 0.9
I ICC values in middle of AIAG’s top 2 classifications:0.96 and 0.995
Part variance values:1, 5, and 25
Laura Lancaster (SAS Institute) Discovery Summit 2015 10 / 39
Simulation Study Design
Three Typical MSA ModelsI Two Factors Crossed (balanced)I Two Factors Nested (balanced)I Three Factors Staggered Nested Design (unbalanced)
Range of bad to good measurement systems (using ICC as themetric)
I ICC values in middle of Wheeler’s EMP classifications:0.1, 0.35, 0.65, 0.9
I ICC values in middle of AIAG’s top 2 classifications:0.96 and 0.995
Part variance values:1, 5, and 25
Laura Lancaster (SAS Institute) Discovery Summit 2015 10 / 39
Simulation Study Design
Three Typical MSA ModelsI Two Factors Crossed (balanced)I Two Factors Nested (balanced)I Three Factors Staggered Nested Design (unbalanced)
Range of bad to good measurement systems (using ICC as themetric)
I ICC values in middle of Wheeler’s EMP classifications:0.1, 0.35, 0.65, 0.9
I ICC values in middle of AIAG’s top 2 classifications:0.96 and 0.995
Part variance values:1, 5, and 25
Laura Lancaster (SAS Institute) Discovery Summit 2015 10 / 39
Simulation Study Design
Three Typical MSA ModelsI Two Factors Crossed (balanced)I Two Factors Nested (balanced)I Three Factors Staggered Nested Design (unbalanced)
Range of bad to good measurement systems (using ICC as themetric)
I ICC values in middle of Wheeler’s EMP classifications:0.1, 0.35, 0.65, 0.9
I ICC values in middle of AIAG’s top 2 classifications:0.96 and 0.995
Part variance values:1, 5, and 25
Laura Lancaster (SAS Institute) Discovery Summit 2015 10 / 39
Simulation Study Design
A given part variance, σ2p, and ICC, ρ, determine the measurement
error variance, σ2e:
σ2e =
σ2p(1− ρ)ρ
Each MSA model uses a weighting schema to set the variancecomponents that contribute to the measurement error.
I Select λ1, λ2, . . . , λm ≥ 0 such that∑m
i=1 λi = 1.I Set σ2
i = λiσ2e for i = 1,2, . . . ,m so that
m∑i=1
σ2i = σ2
e
Laura Lancaster (SAS Institute) Discovery Summit 2015 11 / 39
Simulation Study Design
A given part variance, σ2p, and ICC, ρ, determine the measurement
error variance, σ2e:
σ2e =
σ2p(1− ρ)ρ
Each MSA model uses a weighting schema to set the variancecomponents that contribute to the measurement error.
I Select λ1, λ2, . . . , λm ≥ 0 such that∑m
i=1 λi = 1.I Set σ2
i = λiσ2e for i = 1,2, . . . ,m so that
m∑i=1
σ2i = σ2
e
Laura Lancaster (SAS Institute) Discovery Summit 2015 11 / 39
Simulation Study Design
Scripts written in JSL in JMP Pro 13.
Used the new JMP Pro 13 Simulate function that makes estimatesimulation very easy.Called the Variability platform with following estimation methods:
I REMLI Bayesian (Portnoy-Sahai)I Hybrid (JMP default setting):
If zeroed variance components⇒ Bayesian estimates.Otherwise⇒ REML estimates.
Laura Lancaster (SAS Institute) Discovery Summit 2015 12 / 39
Simulation Study Design
Scripts written in JSL in JMP Pro 13.Used the new JMP Pro 13 Simulate function that makes estimatesimulation very easy.
Called the Variability platform with following estimation methods:I REMLI Bayesian (Portnoy-Sahai)I Hybrid (JMP default setting):
If zeroed variance components⇒ Bayesian estimates.Otherwise⇒ REML estimates.
Laura Lancaster (SAS Institute) Discovery Summit 2015 12 / 39
Simulation Study Design
Scripts written in JSL in JMP Pro 13.Used the new JMP Pro 13 Simulate function that makes estimatesimulation very easy.Called the Variability platform with following estimation methods:
I REMLI Bayesian (Portnoy-Sahai)I Hybrid (JMP default setting):
If zeroed variance components⇒ Bayesian estimates.Otherwise⇒ REML estimates.
Laura Lancaster (SAS Institute) Discovery Summit 2015 12 / 39
Simulation Study Design
Scripts written in JSL in JMP Pro 13.Used the new JMP Pro 13 Simulate function that makes estimatesimulation very easy.Called the Variability platform with following estimation methods:
I REML
I Bayesian (Portnoy-Sahai)I Hybrid (JMP default setting):
If zeroed variance components⇒ Bayesian estimates.Otherwise⇒ REML estimates.
Laura Lancaster (SAS Institute) Discovery Summit 2015 12 / 39
Simulation Study Design
Scripts written in JSL in JMP Pro 13.Used the new JMP Pro 13 Simulate function that makes estimatesimulation very easy.Called the Variability platform with following estimation methods:
I REMLI Bayesian (Portnoy-Sahai)
I Hybrid (JMP default setting):If zeroed variance components⇒ Bayesian estimates.Otherwise⇒ REML estimates.
Laura Lancaster (SAS Institute) Discovery Summit 2015 12 / 39
Simulation Study Design
Scripts written in JSL in JMP Pro 13.Used the new JMP Pro 13 Simulate function that makes estimatesimulation very easy.Called the Variability platform with following estimation methods:
I REMLI Bayesian (Portnoy-Sahai)I Hybrid (JMP default setting):
If zeroed variance components⇒ Bayesian estimates.Otherwise⇒ REML estimates.
Laura Lancaster (SAS Institute) Discovery Summit 2015 12 / 39
Two Factors Crossed Design
Typical MSA design - example from AIAG MSA book3 Operators, 10 Parts, 3 ReplicationsTotal of 90 measurements
500 Simulations
Laura Lancaster (SAS Institute) Discovery Summit 2015 13 / 39
Two Factors Crossed Design
Typical MSA design - example from AIAG MSA book3 Operators, 10 Parts, 3 ReplicationsTotal of 90 measurements
500 Simulations
Laura Lancaster (SAS Institute) Discovery Summit 2015 13 / 39
Two Factors Crossed - Mean Bias
Laura Lancaster (SAS Institute) Discovery Summit 2015 14 / 39
Two Factors Crossed - RMSE
Laura Lancaster (SAS Institute) Discovery Summit 2015 15 / 39
Two Factors Crossed - Standard Deviation
Laura Lancaster (SAS Institute) Discovery Summit 2015 16 / 39
Two Factors Crossed - REML Zeroed
Laura Lancaster (SAS Institute) Discovery Summit 2015 17 / 39
Two Factors Crossed - Wheeler’s EMP Classifications
Laura Lancaster (SAS Institute) Discovery Summit 2015 18 / 39
Two Factors Crossed - AIAG Classifications
Laura Lancaster (SAS Institute) Discovery Summit 2015 19 / 39
Two Factors Crossed - Summary
REML and Hybrid estimates are less biased than Bayesianestimates.
Bayesian estimates have smaller RMSE and standard deviationsthan REML and hybrid estimates.REML zeroing stays about the same for all ICC (good and badsystems) except for part variance.Wheeler’s EMP Classifications - The Bayesian method does aslightly better job. Classifications are best for good systems.AIAG Classifications - The Bayesian method does a slightly betterjob for marginal and acceptable systems.
Laura Lancaster (SAS Institute) Discovery Summit 2015 20 / 39
Two Factors Crossed - Summary
REML and Hybrid estimates are less biased than Bayesianestimates.Bayesian estimates have smaller RMSE and standard deviationsthan REML and hybrid estimates.
REML zeroing stays about the same for all ICC (good and badsystems) except for part variance.Wheeler’s EMP Classifications - The Bayesian method does aslightly better job. Classifications are best for good systems.AIAG Classifications - The Bayesian method does a slightly betterjob for marginal and acceptable systems.
Laura Lancaster (SAS Institute) Discovery Summit 2015 20 / 39
Two Factors Crossed - Summary
REML and Hybrid estimates are less biased than Bayesianestimates.Bayesian estimates have smaller RMSE and standard deviationsthan REML and hybrid estimates.REML zeroing stays about the same for all ICC (good and badsystems) except for part variance.
Wheeler’s EMP Classifications - The Bayesian method does aslightly better job. Classifications are best for good systems.AIAG Classifications - The Bayesian method does a slightly betterjob for marginal and acceptable systems.
Laura Lancaster (SAS Institute) Discovery Summit 2015 20 / 39
Two Factors Crossed - Summary
REML and Hybrid estimates are less biased than Bayesianestimates.Bayesian estimates have smaller RMSE and standard deviationsthan REML and hybrid estimates.REML zeroing stays about the same for all ICC (good and badsystems) except for part variance.Wheeler’s EMP Classifications - The Bayesian method does aslightly better job. Classifications are best for good systems.
AIAG Classifications - The Bayesian method does a slightly betterjob for marginal and acceptable systems.
Laura Lancaster (SAS Institute) Discovery Summit 2015 20 / 39
Two Factors Crossed - Summary
REML and Hybrid estimates are less biased than Bayesianestimates.Bayesian estimates have smaller RMSE and standard deviationsthan REML and hybrid estimates.REML zeroing stays about the same for all ICC (good and badsystems) except for part variance.Wheeler’s EMP Classifications - The Bayesian method does aslightly better job. Classifications are best for good systems.AIAG Classifications - The Bayesian method does a slightly betterjob for marginal and acceptable systems.
Laura Lancaster (SAS Institute) Discovery Summit 2015 20 / 39
Two Factors Nested Design
Typical MSA design - example from Montgomery and Runger(1993)3 Operators, 20 Parts, 2 ReplicationsTotal of 120 measurements
500 Simulations
Laura Lancaster (SAS Institute) Discovery Summit 2015 21 / 39
Two Factors Nested Design
Typical MSA design - example from Montgomery and Runger(1993)3 Operators, 20 Parts, 2 ReplicationsTotal of 120 measurements
500 Simulations
Laura Lancaster (SAS Institute) Discovery Summit 2015 21 / 39
Two Factors Nested - Mean Bias
Laura Lancaster (SAS Institute) Discovery Summit 2015 22 / 39
Two Factors Nested - RMSE
Laura Lancaster (SAS Institute) Discovery Summit 2015 23 / 39
Two Factors Nested - Standard Deviation
Laura Lancaster (SAS Institute) Discovery Summit 2015 24 / 39
Two Factors Nested - REML Zeroed
Laura Lancaster (SAS Institute) Discovery Summit 2015 25 / 39
Two Factors Nested - Wheeler’s EMP Classifications
Laura Lancaster (SAS Institute) Discovery Summit 2015 26 / 39
Two Factors Nested - AIAG Classifications
Laura Lancaster (SAS Institute) Discovery Summit 2015 27 / 39
Two Factors Nested - Summary
REML and hybrid estimates are less biased than Bayesianestimates.
Bayesian estimates have smaller RMSE and standard deviationsthan REML and hybrid estimates.Wheeler’s EMP Classifications -
I REML and Hybrid methods do a slightly better job classifying badMSA systems, but the Bayesian method does a better job for bettersystems.
I Classifications are more accurate for good systems in general.AIAG Classifications -
I The Bayesian method classifies better for marginal systems butclassifies horribly for acceptable systems (100% wrong).
I REML classifies slightly worse for marginal systems but classifiesmuch better than Bayesian and Hybrid for acceptable systems.
I Overall, the accuracy of the classifications for acceptable systemsis not good.
Laura Lancaster (SAS Institute) Discovery Summit 2015 28 / 39
Two Factors Nested - Summary
REML and hybrid estimates are less biased than Bayesianestimates.Bayesian estimates have smaller RMSE and standard deviationsthan REML and hybrid estimates.
Wheeler’s EMP Classifications -I REML and Hybrid methods do a slightly better job classifying bad
MSA systems, but the Bayesian method does a better job for bettersystems.
I Classifications are more accurate for good systems in general.AIAG Classifications -
I The Bayesian method classifies better for marginal systems butclassifies horribly for acceptable systems (100% wrong).
I REML classifies slightly worse for marginal systems but classifiesmuch better than Bayesian and Hybrid for acceptable systems.
I Overall, the accuracy of the classifications for acceptable systemsis not good.
Laura Lancaster (SAS Institute) Discovery Summit 2015 28 / 39
Two Factors Nested - Summary
REML and hybrid estimates are less biased than Bayesianestimates.Bayesian estimates have smaller RMSE and standard deviationsthan REML and hybrid estimates.Wheeler’s EMP Classifications -
I REML and Hybrid methods do a slightly better job classifying badMSA systems, but the Bayesian method does a better job for bettersystems.
I Classifications are more accurate for good systems in general.AIAG Classifications -
I The Bayesian method classifies better for marginal systems butclassifies horribly for acceptable systems (100% wrong).
I REML classifies slightly worse for marginal systems but classifiesmuch better than Bayesian and Hybrid for acceptable systems.
I Overall, the accuracy of the classifications for acceptable systemsis not good.
Laura Lancaster (SAS Institute) Discovery Summit 2015 28 / 39
Two Factors Nested - Summary
REML and hybrid estimates are less biased than Bayesianestimates.Bayesian estimates have smaller RMSE and standard deviationsthan REML and hybrid estimates.Wheeler’s EMP Classifications -
I REML and Hybrid methods do a slightly better job classifying badMSA systems, but the Bayesian method does a better job for bettersystems.
I Classifications are more accurate for good systems in general.
AIAG Classifications -I The Bayesian method classifies better for marginal systems but
classifies horribly for acceptable systems (100% wrong).I REML classifies slightly worse for marginal systems but classifies
much better than Bayesian and Hybrid for acceptable systems.I Overall, the accuracy of the classifications for acceptable systems
is not good.
Laura Lancaster (SAS Institute) Discovery Summit 2015 28 / 39
Two Factors Nested - Summary
REML and hybrid estimates are less biased than Bayesianestimates.Bayesian estimates have smaller RMSE and standard deviationsthan REML and hybrid estimates.Wheeler’s EMP Classifications -
I REML and Hybrid methods do a slightly better job classifying badMSA systems, but the Bayesian method does a better job for bettersystems.
I Classifications are more accurate for good systems in general.AIAG Classifications -
I The Bayesian method classifies better for marginal systems butclassifies horribly for acceptable systems (100% wrong).
I REML classifies slightly worse for marginal systems but classifiesmuch better than Bayesian and Hybrid for acceptable systems.
I Overall, the accuracy of the classifications for acceptable systemsis not good.
Laura Lancaster (SAS Institute) Discovery Summit 2015 28 / 39
Two Factors Nested - Summary
REML and hybrid estimates are less biased than Bayesianestimates.Bayesian estimates have smaller RMSE and standard deviationsthan REML and hybrid estimates.Wheeler’s EMP Classifications -
I REML and Hybrid methods do a slightly better job classifying badMSA systems, but the Bayesian method does a better job for bettersystems.
I Classifications are more accurate for good systems in general.AIAG Classifications -
I The Bayesian method classifies better for marginal systems butclassifies horribly for acceptable systems (100% wrong).
I REML classifies slightly worse for marginal systems but classifiesmuch better than Bayesian and Hybrid for acceptable systems.
I Overall, the accuracy of the classifications for acceptable systemsis not good.
Laura Lancaster (SAS Institute) Discovery Summit 2015 28 / 39
Two Factors Nested - Summary
REML and hybrid estimates are less biased than Bayesianestimates.Bayesian estimates have smaller RMSE and standard deviationsthan REML and hybrid estimates.Wheeler’s EMP Classifications -
I REML and Hybrid methods do a slightly better job classifying badMSA systems, but the Bayesian method does a better job for bettersystems.
I Classifications are more accurate for good systems in general.AIAG Classifications -
I The Bayesian method classifies better for marginal systems butclassifies horribly for acceptable systems (100% wrong).
I REML classifies slightly worse for marginal systems but classifiesmuch better than Bayesian and Hybrid for acceptable systems.
I Overall, the accuracy of the classifications for acceptable systemsis not good.
Laura Lancaster (SAS Institute) Discovery Summit 2015 28 / 39
Three Factors Staggered Nested Design
Polyethylene pellets MSA design example from Lawson (2008)I 30 lotsI 2 boxes from each lotI 2 preparations within box 1, 1 preparation within box 2I 2 measurements from preparation 1 within box 1, 1 measurement
from preparation 2 within box 1, 1 measurement from preparation 3within box 2.
I Total of 120 measurements of tensile strengthI Highly unbalanced
Laura Lancaster (SAS Institute) Discovery Summit 2015 29 / 39
Three Factors Staggered Nested - Mean Bias
Laura Lancaster (SAS Institute) Discovery Summit 2015 30 / 39
Three Factors Staggered Nested - RMSE
Laura Lancaster (SAS Institute) Discovery Summit 2015 31 / 39
Three Factors Staggered Nested - Standard Deviation
Laura Lancaster (SAS Institute) Discovery Summit 2015 32 / 39
Three Factors Staggered Nested - REML Zeroed
Laura Lancaster (SAS Institute) Discovery Summit 2015 33 / 39
Three Factors Staggered Nested - Wheeler’s EMPClassifications
Laura Lancaster (SAS Institute) Discovery Summit 2015 34 / 39
Three Factors Staggered Nested - AIAGClassifications
Laura Lancaster (SAS Institute) Discovery Summit 2015 35 / 39
Three Factors Staggered Nested - Summary
REML and hybrid estimates are less biased than Bayesianestimates.
Bayesian estimates have smaller RMSE than REML and hybridestimates for worse systems, but they are larger for bettersystems.Bayesian estimates have smaller standard deviations than REMLand hybrid estimates.REML zeroing increases as ICC increases.Wheeler’s EMP Classifications -
I Bayesian method does a slightly better job classifying bad MSAsystems.
I REML does a better job classifying good systems where Bayesianand Hybrid methods classify quite poorly.
AIAG Classifications -I Bayesian and Hybrid methods do a poor job classifying marginal
and acceptable systems (almost always incorrect).I REML does a better job but is still wrong more than 60% of time!
Laura Lancaster (SAS Institute) Discovery Summit 2015 36 / 39
Three Factors Staggered Nested - Summary
REML and hybrid estimates are less biased than Bayesianestimates.Bayesian estimates have smaller RMSE than REML and hybridestimates for worse systems, but they are larger for bettersystems.
Bayesian estimates have smaller standard deviations than REMLand hybrid estimates.REML zeroing increases as ICC increases.Wheeler’s EMP Classifications -
I Bayesian method does a slightly better job classifying bad MSAsystems.
I REML does a better job classifying good systems where Bayesianand Hybrid methods classify quite poorly.
AIAG Classifications -I Bayesian and Hybrid methods do a poor job classifying marginal
and acceptable systems (almost always incorrect).I REML does a better job but is still wrong more than 60% of time!
Laura Lancaster (SAS Institute) Discovery Summit 2015 36 / 39
Three Factors Staggered Nested - Summary
REML and hybrid estimates are less biased than Bayesianestimates.Bayesian estimates have smaller RMSE than REML and hybridestimates for worse systems, but they are larger for bettersystems.Bayesian estimates have smaller standard deviations than REMLand hybrid estimates.
REML zeroing increases as ICC increases.Wheeler’s EMP Classifications -
I Bayesian method does a slightly better job classifying bad MSAsystems.
I REML does a better job classifying good systems where Bayesianand Hybrid methods classify quite poorly.
AIAG Classifications -I Bayesian and Hybrid methods do a poor job classifying marginal
and acceptable systems (almost always incorrect).I REML does a better job but is still wrong more than 60% of time!
Laura Lancaster (SAS Institute) Discovery Summit 2015 36 / 39
Three Factors Staggered Nested - Summary
REML and hybrid estimates are less biased than Bayesianestimates.Bayesian estimates have smaller RMSE than REML and hybridestimates for worse systems, but they are larger for bettersystems.Bayesian estimates have smaller standard deviations than REMLand hybrid estimates.REML zeroing increases as ICC increases.
Wheeler’s EMP Classifications -I Bayesian method does a slightly better job classifying bad MSA
systems.I REML does a better job classifying good systems where Bayesian
and Hybrid methods classify quite poorly.AIAG Classifications -
I Bayesian and Hybrid methods do a poor job classifying marginaland acceptable systems (almost always incorrect).
I REML does a better job but is still wrong more than 60% of time!
Laura Lancaster (SAS Institute) Discovery Summit 2015 36 / 39
Three Factors Staggered Nested - Summary
REML and hybrid estimates are less biased than Bayesianestimates.Bayesian estimates have smaller RMSE than REML and hybridestimates for worse systems, but they are larger for bettersystems.Bayesian estimates have smaller standard deviations than REMLand hybrid estimates.REML zeroing increases as ICC increases.Wheeler’s EMP Classifications -
I Bayesian method does a slightly better job classifying bad MSAsystems.
I REML does a better job classifying good systems where Bayesianand Hybrid methods classify quite poorly.
AIAG Classifications -I Bayesian and Hybrid methods do a poor job classifying marginal
and acceptable systems (almost always incorrect).I REML does a better job but is still wrong more than 60% of time!
Laura Lancaster (SAS Institute) Discovery Summit 2015 36 / 39
Three Factors Staggered Nested - Summary
REML and hybrid estimates are less biased than Bayesianestimates.Bayesian estimates have smaller RMSE than REML and hybridestimates for worse systems, but they are larger for bettersystems.Bayesian estimates have smaller standard deviations than REMLand hybrid estimates.REML zeroing increases as ICC increases.Wheeler’s EMP Classifications -
I Bayesian method does a slightly better job classifying bad MSAsystems.
I REML does a better job classifying good systems where Bayesianand Hybrid methods classify quite poorly.
AIAG Classifications -I Bayesian and Hybrid methods do a poor job classifying marginal
and acceptable systems (almost always incorrect).I REML does a better job but is still wrong more than 60% of time!
Laura Lancaster (SAS Institute) Discovery Summit 2015 36 / 39
Three Factors Staggered Nested - Summary
REML and hybrid estimates are less biased than Bayesianestimates.Bayesian estimates have smaller RMSE than REML and hybridestimates for worse systems, but they are larger for bettersystems.Bayesian estimates have smaller standard deviations than REMLand hybrid estimates.REML zeroing increases as ICC increases.Wheeler’s EMP Classifications -
I Bayesian method does a slightly better job classifying bad MSAsystems.
I REML does a better job classifying good systems where Bayesianand Hybrid methods classify quite poorly.
AIAG Classifications -I Bayesian and Hybrid methods do a poor job classifying marginal
and acceptable systems (almost always incorrect).
I REML does a better job but is still wrong more than 60% of time!
Laura Lancaster (SAS Institute) Discovery Summit 2015 36 / 39
Three Factors Staggered Nested - Summary
REML and hybrid estimates are less biased than Bayesianestimates.Bayesian estimates have smaller RMSE than REML and hybridestimates for worse systems, but they are larger for bettersystems.Bayesian estimates have smaller standard deviations than REMLand hybrid estimates.REML zeroing increases as ICC increases.Wheeler’s EMP Classifications -
I Bayesian method does a slightly better job classifying bad MSAsystems.
I REML does a better job classifying good systems where Bayesianand Hybrid methods classify quite poorly.
AIAG Classifications -I Bayesian and Hybrid methods do a poor job classifying marginal
and acceptable systems (almost always incorrect).I REML does a better job but is still wrong more than 60% of time!
Laura Lancaster (SAS Institute) Discovery Summit 2015 36 / 39
Conclusions
REML and Hybrid estimates have less mean bias than Bayesianestimates.
Bayesian estimates usually have smaller RMSE than REML andHybrid estimates.Bayesian estimates have smaller standard deviations than REMLand Hybrid estimates.We recommend not using staggered nested designs when at allpossible, especially if AIAG standards are required.
I It will be very difficult to classify your system as marginal oracceptable.
I If this design must be used, we recommend using REML estimates.Otherwise, we think the hybrid method is a good strategy for TwoFactors Crossed and Two Factors Nested designs.
I Its performance is a usually a good compromise.I Exception: Two Factors Nested designs when trying to prove a
system is acceptable by AIAG standards.
Laura Lancaster (SAS Institute) Discovery Summit 2015 37 / 39
Conclusions
REML and Hybrid estimates have less mean bias than Bayesianestimates.Bayesian estimates usually have smaller RMSE than REML andHybrid estimates.
Bayesian estimates have smaller standard deviations than REMLand Hybrid estimates.We recommend not using staggered nested designs when at allpossible, especially if AIAG standards are required.
I It will be very difficult to classify your system as marginal oracceptable.
I If this design must be used, we recommend using REML estimates.Otherwise, we think the hybrid method is a good strategy for TwoFactors Crossed and Two Factors Nested designs.
I Its performance is a usually a good compromise.I Exception: Two Factors Nested designs when trying to prove a
system is acceptable by AIAG standards.
Laura Lancaster (SAS Institute) Discovery Summit 2015 37 / 39
Conclusions
REML and Hybrid estimates have less mean bias than Bayesianestimates.Bayesian estimates usually have smaller RMSE than REML andHybrid estimates.Bayesian estimates have smaller standard deviations than REMLand Hybrid estimates.
We recommend not using staggered nested designs when at allpossible, especially if AIAG standards are required.
I It will be very difficult to classify your system as marginal oracceptable.
I If this design must be used, we recommend using REML estimates.Otherwise, we think the hybrid method is a good strategy for TwoFactors Crossed and Two Factors Nested designs.
I Its performance is a usually a good compromise.I Exception: Two Factors Nested designs when trying to prove a
system is acceptable by AIAG standards.
Laura Lancaster (SAS Institute) Discovery Summit 2015 37 / 39
Conclusions
REML and Hybrid estimates have less mean bias than Bayesianestimates.Bayesian estimates usually have smaller RMSE than REML andHybrid estimates.Bayesian estimates have smaller standard deviations than REMLand Hybrid estimates.We recommend not using staggered nested designs when at allpossible, especially if AIAG standards are required.
I It will be very difficult to classify your system as marginal oracceptable.
I If this design must be used, we recommend using REML estimates.Otherwise, we think the hybrid method is a good strategy for TwoFactors Crossed and Two Factors Nested designs.
I Its performance is a usually a good compromise.I Exception: Two Factors Nested designs when trying to prove a
system is acceptable by AIAG standards.
Laura Lancaster (SAS Institute) Discovery Summit 2015 37 / 39
Conclusions
REML and Hybrid estimates have less mean bias than Bayesianestimates.Bayesian estimates usually have smaller RMSE than REML andHybrid estimates.Bayesian estimates have smaller standard deviations than REMLand Hybrid estimates.We recommend not using staggered nested designs when at allpossible, especially if AIAG standards are required.
I It will be very difficult to classify your system as marginal oracceptable.
I If this design must be used, we recommend using REML estimates.Otherwise, we think the hybrid method is a good strategy for TwoFactors Crossed and Two Factors Nested designs.
I Its performance is a usually a good compromise.I Exception: Two Factors Nested designs when trying to prove a
system is acceptable by AIAG standards.
Laura Lancaster (SAS Institute) Discovery Summit 2015 37 / 39
Conclusions
REML and Hybrid estimates have less mean bias than Bayesianestimates.Bayesian estimates usually have smaller RMSE than REML andHybrid estimates.Bayesian estimates have smaller standard deviations than REMLand Hybrid estimates.We recommend not using staggered nested designs when at allpossible, especially if AIAG standards are required.
I It will be very difficult to classify your system as marginal oracceptable.
I If this design must be used, we recommend using REML estimates.
Otherwise, we think the hybrid method is a good strategy for TwoFactors Crossed and Two Factors Nested designs.
I Its performance is a usually a good compromise.I Exception: Two Factors Nested designs when trying to prove a
system is acceptable by AIAG standards.
Laura Lancaster (SAS Institute) Discovery Summit 2015 37 / 39
Conclusions
REML and Hybrid estimates have less mean bias than Bayesianestimates.Bayesian estimates usually have smaller RMSE than REML andHybrid estimates.Bayesian estimates have smaller standard deviations than REMLand Hybrid estimates.We recommend not using staggered nested designs when at allpossible, especially if AIAG standards are required.
I It will be very difficult to classify your system as marginal oracceptable.
I If this design must be used, we recommend using REML estimates.Otherwise, we think the hybrid method is a good strategy for TwoFactors Crossed and Two Factors Nested designs.
I Its performance is a usually a good compromise.I Exception: Two Factors Nested designs when trying to prove a
system is acceptable by AIAG standards.
Laura Lancaster (SAS Institute) Discovery Summit 2015 37 / 39
Conclusions
REML and Hybrid estimates have less mean bias than Bayesianestimates.Bayesian estimates usually have smaller RMSE than REML andHybrid estimates.Bayesian estimates have smaller standard deviations than REMLand Hybrid estimates.We recommend not using staggered nested designs when at allpossible, especially if AIAG standards are required.
I It will be very difficult to classify your system as marginal oracceptable.
I If this design must be used, we recommend using REML estimates.Otherwise, we think the hybrid method is a good strategy for TwoFactors Crossed and Two Factors Nested designs.
I Its performance is a usually a good compromise.
I Exception: Two Factors Nested designs when trying to prove asystem is acceptable by AIAG standards.
Laura Lancaster (SAS Institute) Discovery Summit 2015 37 / 39
Conclusions
REML and Hybrid estimates have less mean bias than Bayesianestimates.Bayesian estimates usually have smaller RMSE than REML andHybrid estimates.Bayesian estimates have smaller standard deviations than REMLand Hybrid estimates.We recommend not using staggered nested designs when at allpossible, especially if AIAG standards are required.
I It will be very difficult to classify your system as marginal oracceptable.
I If this design must be used, we recommend using REML estimates.Otherwise, we think the hybrid method is a good strategy for TwoFactors Crossed and Two Factors Nested designs.
I Its performance is a usually a good compromise.I Exception: Two Factors Nested designs when trying to prove a
system is acceptable by AIAG standards.
Laura Lancaster (SAS Institute) Discovery Summit 2015 37 / 39
References
Automotive Industry Action Group (2002), Measurement SystemsAnalysis Reference Manual, 3rd Edition.Lawson (2008), “Bayesian Interval Estimates of VarianceComponents Used in Quality Improvement Studies, ” QualityEngineering, 20, 334-345.Montgomery and Runger (1993), “Gauge Capability Analysis andDesigned Experiments. Part II: Experimental Design Models andVariance Component Estimation,” Quality Engineering, 6(2),289-305.Portnoy (1971), “Formal Bayes Estimation With Application To aRandom Effect Model,” Annals Of Mathematical Statistics, 42,1379-1402.Sahai (1974), “Some Formal Bayes Estimators of VarianceComponents in Balanced Three-Stage Nested Random EffectsModel,” Communications in Statistics, 3, 233-242.
Laura Lancaster (SAS Institute) Discovery Summit 2015 38 / 39