Date post: | 16-Jan-2016 |
Category: |
Documents |
Upload: | violet-lang |
View: | 214 times |
Download: | 0 times |
Company Confidential
Registration Management Committee (RMC)
1
2010 Report on Oasis Data for2007-2010
Primary focus on 12 month roll starting in
November 2009 to December 2010(For Open Meeting)
San Diego, CAJanuary 18 – 19, 2011
R. Darrell TaylorRaytheon
San Diego, CAJanuary 18-19, 2011
Registration Management Committee (RMC)
2
Executive Summary
• Setting a Baseline for the future
• Statistically Based Analysis
• Two-fold Analysis of data
– Trends for the total population
– Trends for the individual CB’s
San Diego, CAJanuary 18-19, 2011
Registration Management Committee (RMC)
3
Agenda
• OASIS Entries – Audits
• OASIS Nonconformities
• AS9100 Elements – Quick Overview no change
• Feedback
• Monitoring Model
San Diego, CAJanuary 18-19, 2011
Registration Management Committee (RMC)
4
Comparison of Entries 2007 -2010
• Entries:
– 2007 – 5114
– 2008 – 5972 ^14%
– 2009 – 6969 ^17%
– 2010 – 5987
» As of 12/2010
5114
5972
6969
5987
0
1000
2000
3000
4000
5000
6000
7000
2007 2008 2009 2010 YTD
Total Entries
Consistent with Record Entry Delays
Projecting a flat to a slight increase
compared to 2009 year
San Diego, CAJanuary 18-19, 2011
Registration Management Committee (RMC)
5
Comparison of Types2007 -2010
2007 3364 888 862
2008 3977 923 1072
2009 4421 1486 1062
2010 4982 1426 823
SUR REC INT
SUR=Surveillance
REC = Recertification
INT = Initial
NOTE : 2010 is still not complete, consistent with previous years
2007 0.66 0.17 0.172008 0.67 0.15 0.182009 0.63 0.21 0.152010 0.69 0.20 0.11
Percentages
San Diego, CAJanuary 18-19, 2011
Registration Management Committee (RMC)
6
Standards Entered Into Database
0.0000.0050.0100.0150.0200.0250.0300.035
0.0400.0450.050
2007 2008 2009 2010
AS9120AS9110NBR15100:2004EN9100:2003/ S1EN9120:P2/ S1EN9100:P2/ S1AS9100C
AS9100B:2007- 95%2008 - 95%2009 - 94%2010 -92%
2 AS9100C audits now in the database
San Diego, CAJanuary 18-19, 2011
Registration Management Committee (RMC)
7
Summary of OASIS Data
0
0.5
1
1.5
2
2.5
3
3.5
2008 2009 2010
Average # of N/ C per audit
0
0.1
0.2
0.3
2008 2009 2010
2.2
2.4
2.6
2.8
2008 2009 2010
MAJORS
MINORS
Average number of Non Conformities per audit slightly over 3Average number of minors 2.4 per audit
On average a major is written every 5th audit
Minors – 15,814
Majors - 1117
San Diego, CAJanuary 18-19, 2011
Registration Management Committee (RMC)
8
AS9100 Nonconformities Top 10
Top Ten 2010 same as 2009 & 2008
0 500 1000 1500 2000 2500 3000
4.3 Conf iguration Management
6.3 Inf ras truc ture
5.4 Planning
4.1 QMS - General Requirements
7.3 Des ign & Dev elopment
7.2 Cus tomer Related Proc es s es
5.6 Management Rev iew
6.2 - Human Res ourc es
8.3 Control of Non-Conf orming Produc t
7.6 Control of Monitor ing and Meas ur ing Equipment
8.5 - Improv ement
8.2- Monitor ing & Meas urement
7.4 - Purc has ing
4.2 Doc umentation Requirements
7.5 - Produc tion and Serv ic e Prov is ion
M ajor
M inor
Will monitor as we transition to AS9100C and determine
if there are any industry related trends
San Diego, CAJanuary 18-19, 2011
Registration Management Committee (RMC)
9
Feedback Analysis
2009 average 112 per month, none in December2008 – average 110 per month as of August
0
500
1000
1500
2000
2500
2007 2008 2009 2010YTD
OPENCLOSED
San Diego, CAJanuary 18-19, 2011
Registration Management Committee (RMC)
10
ANAB Audit Summary
• 2010
– 6.07 Minors per audit, .4 Majors per audit
• Other Categories monitored:
– Escalated N/C Customer Complaints
– Negative Shift Suspensions (2 Yrs)
– Additional Audits
– N/C from outside assessments
San Diego, CAJanuary 18-19, 2011
Registration Management Committee (RMC)
11
Certification Body Statistical Assessment Model
ANAB Audits - 6 Criteria
OASIS Results - 3 Criteria
Feedback - 2 Criteria
General - 1 Criteria
+ 1
Added Old Feedback “OPEN” as separate criteria
ANAB Performance - + 4
Customer Complaints
Suspensions
C/A’s outside of Audits
Extra Audits
San Diego, CAJanuary 18-19, 2011
Registration Management Committee (RMC)
12
Threshold Violations
0
2
4
6
8
10
12
14
16
Violations
• RMC & ANAB need to research
– >6
• ANAB to report to RMC
– 4-5
• ANAB to investigate
– 2-3
• No Action
– <2
San Diego, CAJanuary 18-19, 2011
Registration Management Committee (RMC)
13
Quantity of CB’s & Threshold Violations 11/2009 – 11/2010
6
16
10
7
0
2
4
6
8
10
12
14
16
<2 2 & 3 4 & 5 >5
Threshold Violations
San Diego, CAJanuary 18-19, 2011
Registration Management Committee (RMC)
14
Non - Statistical Driven Metrics
• Feedback (Only 9 CB’s did not have a hit)
– Closed > average 30 Days
– Open > average 30 Days
– OPEN over one year
• Escalated N/C’s (9 CB’s had escalated N/C’s)
San Diego, CAJanuary 18-19, 2011
Registration Management Committee (RMC)
15
Questions
Company Confidential
Registration Management Committee (RMC)
16
Back Up
San Diego, CAJanuary 18-19, 2011
Registration Management Committee (RMC)
17
ANAB Audits - Minors
108642
Median
Mean
6.56.05.55.04.5
1st Quartile 3.5000Median 6.00003rd Quartile 7.0000Maximum 11.0000
4.7339 6.4013
5.0000 6.0000
2.0334 3.2480
A-Squared 0.74P-Value 0.050
Mean 5.5676StDev 2.5005Variance 6.2523Skewness 0.140856Kurtosis -0.657909N 37
Minimum 2.0000
Anderson-Darling Normality Test
95% Confidence I nterval for Mean
95% Confidence I nterval for Median
95% Confidence I nterval for StDev95% Confidence I ntervals
Summary for Minors
6543210
99
95
90
80
70
605040
30
20
10
5
1
Ratio Minors
Pe
rce
nt
Mean 2.552StDev 1.275N 37RJ 0.986P-Value >0.100
Probability Plot of Ratio MinorsNormal
Normal Distribution
Identify those CB’s in the top 5 % receiving Minors from
ANAB audits
San Diego, CAJanuary 18-19, 2011
Registration Management Committee (RMC)
18
ANAB Audits - Majors
NOT Normal Distribution
2.01.51.00.50.0
Median
Mean
0.50.40.30.20.10.0
1st Quartile 0.00000
Median 0.000003rd Quartile 0.50000Maximum 2.00000
0.16380 0.49837
0.00000 0.33333
0.40801 0.65172
A-Squared 4.09P-Value < 0.005
Mean 0.33108StDev 0.50173Variance 0.25173
Skewness 1.73203Kurtosis 2.69073N 37
Minimum 0.00000
Anderson-Darling Normality Test
95% Confidence I nterval for Mean
95% Confidence I nterval for Median
95% Confidence I nterval for StDev95% Confidence I ntervals
Summary for Ratio Majors
2.01.51.00.50.0-0.5-1.0
99
95
90
80
70
605040
30
20
10
5
1
Ratio Majors
Pe
rce
nt
Mean 0.3311StDev 0.5017N 37RJ 0.966P-Value 0.040
Probability Plot of Ratio MajorsNormal
Identify those CB’s in the top 5 % receiving Majors from
ANAB audits
San Diego, CAJanuary 18-19, 2011
Registration Management Committee (RMC)
19
ANAB Audits – Majors (cont.)
Poisson Distribution supports the use of > 2 as threshold
2.01.51.00.50.0
Median
Mean
0.50.40.30.20.10.0
1st Quartile 0.00000
Median 0.000003rd Quartile 0.50000Maximum 2.00000
0.16380 0.49837
0.00000 0.33333
0.40801 0.65172
A-Squared 4.09P-Value < 0.005
Mean 0.33108StDev 0.50173Variance 0.25173
Skewness 1.73203Kurtosis 2.69073N 37
Minimum 0.00000
Anderson-Darling Normality Test
95% Confidence I nterval for Mean
95% Confidence I nterval for Median
95% Confidence I nterval for StDev95% Confidence I ntervals
Summary for Ratio Majors
0.5
0.4
0.3
0.2
0.1
0.0
X
Pro
ba
bili
ty
3
0.0414
0
Distribution PlotPoisson, Mean=0.757
San Diego, CAJanuary 18-19, 2011
Registration Management Committee (RMC)
20
ANAB Audits – Ratio Total
Normal Distribution
6543210
99
95
90
80
70
605040
30
20
10
5
1
Ratio TotalP
erc
en
t
Mean 2.883StDev 1.360N 37RJ 0.992P-Value >0.100
Probability Plot of Ratio TotalNormal
6.44.83.21.6
Median
Mean
3.503.253.002.752.502.252.00
1st Quartile 2.0000
Median 3.00003rd Quartile 3.5000Maximum 6.0000
2.4294 3.3364
2.0336 3.4832
1.1061 1.7668
A-Squared 0.42P-Value 0.302
Mean 2.8829StDev 1.3602Variance 1.8501
Skewness 0.309638Kurtosis -0.423729N 37
Minimum 0.6667
Anderson-Darling Normality Test
95% Confidence I nterval for Mean
95% Confidence I nterval for Median
95% Confidence I nterval for StDev95% Confidence I ntervals
Summary for Ratio Total
Identify those CB’s in the top 5 % receiving
Nonconformities as a ratio to audits performed from ANAB
audits
San Diego, CAJanuary 18-19, 2011
Registration Management Committee (RMC)
21
ANAB Audits – Ratio Minors
Normal Distribution
4.83.62.41.2
Median
Mean
3.02.82.62.42.22.0
1st Quartile 1.2917
Median 3.00003rd Quartile 3.5000Maximum 5.5000
2.1266 2.9770
2.0000 3.0000
1.0371 1.6565
A-Squared 0.79P-Value 0.037
Mean 2.5518StDev 1.2753Variance 1.6263
Skewness 0.290702Kurtosis -0.575196N 37
Minimum 0.6667
Anderson-Darling Normality Test
95% Confidence I nterval for Mean
95% Confidence I nterval for Median
95% Confidence I nterval for StDev95% Confidence I ntervals
Summary for Ratio Minors
6543210
99
95
90
80
70
605040
30
20
10
5
1
Ratio Minors
Pe
rce
nt
Mean 2.552StDev 1.275N 37RJ 0.986P-Value >0.100
Probability Plot of Ratio MinorsNormal
Identify those CB’s in the top 5 % receiving Minors as a ratio to audits performed
from ANAB audits
San Diego, CAJanuary 18-19, 2011
Registration Management Committee (RMC)
22
ANAB Audits – Ratio Majors
NOT Normal Distribution
2.01.51.00.50.0
Median
Mean
0.50.40.30.20.10.0
1st Quartile 0.00000
Median 0.000003rd Quartile 0.50000Maximum 2.00000
0.16380 0.49837
0.00000 0.33333
0.40801 0.65172
A-Squared 4.09P-Value < 0.005
Mean 0.33108StDev 0.50173Variance 0.25173
Skewness 1.73203Kurtosis 2.69073N 37
Minimum 0.00000
Anderson-Darling Normality Test
95% Confidence I nterval for Mean
95% Confidence I nterval for Median
95% Confidence I nterval for StDev95% Confidence I ntervals
Summary for Ratio Majors
2.01.51.00.50.0-0.5-1.0
99
95
90
80
70
605040
30
20
10
5
1
Ratio Majors
Pe
rce
nt
Mean 0.3311StDev 0.5017N 37RJ 0.966P-Value 0.040
Probability Plot of Ratio MajorsNormal
Identify those CB’s in the top 5 % receiving Majors as a ratio to audits performed
from ANAB audits
San Diego, CAJanuary 18-19, 2011
Registration Management Committee (RMC)
23
ANAB Audits – Ratio Majors (cont.)
2.01.51.00.50.0
Median
Mean
0.50.40.30.20.10.0
1st Quartile 0.00000
Median 0.000003rd Quartile 0.50000Maximum 2.00000
0.16380 0.49837
0.00000 0.33333
0.40801 0.65172
A-Squared 4.09P-Value < 0.005
Mean 0.33108StDev 0.50173Variance 0.25173
Skewness 1.73203Kurtosis 2.69073N 37
Minimum 0.00000
Anderson-Darling Normality Test
95% Confidence I nterval for Mean
95% Confidence I nterval for Median
95% Confidence I nterval for StDev95% Confidence I ntervals
Summary for Ratio Majors
Poisson Distribution supports the use of >1
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0
X
Pro
ba
bili
ty2
0.04410
Distribution PlotPoisson, Mean=0.33108
San Diego, CAJanuary 18-19, 2011
Registration Management Committee (RMC)
24
ANAB Score Negative Shift
• Formula TY Ratio of Total – LY Ratio of Total
– Trigger count if decrease is > than 1 SD (TY)
• Based on a one standard deviation shift in the negative direction, may be an indicator of a potential issue
TY= This Year
LY = Last Year
SD= Standard Deviation
Identify those CB’s with a greater than one standard
deviation shift in performance in the negative
direction
San Diego, CAJanuary 18-19, 2011
Registration Management Committee (RMC)
25
FEEDBACK
• Rule of Thumb threshold > 30 days based on “Closed”
• Rule of Thumb threshold > 30 days based on “Currently OPEN”
• Rule of Thumb any violation of older “OPEN” feedback
Identify CB’s with outstanding feedback responses >30 days
San Diego, CAJanuary 18-19, 2011
Registration Management Committee (RMC)
26
ANAB DATA – CB Performance
• Additional ANAB Audits >0
• Escalated Nonconformities > 0
• Customer Complaints > 2 over two years
• Corrective Actions issued outside of Assessment Minors >1, Majors >0
• Suspensions in past 2 years >0
• Outside Assessment Influence
San Diego, CAJanuary 18-19, 2011
Registration Management Committee (RMC)
27
OASIS Ratio of Total Findings Modified
Modified criteria excluding outlier
543210
99
95
90
80
70
605040
30
20
10
5
1
Ratio Total_OP
erc
en
t
Mean 2.477StDev 0.9799N 36RJ 0.981P-Value >0.100
Probability Plot of Ratio Total_ONormal
54321
Median
Mean
2.82.62.42.22.0
1st Quartile 1.7699Median 2.39053rd Quartile 3.0867Maximum 5.0000
2.1456 2.8086
1.9483 2.7283
0.7948 1.2782
A-Squared 0.39P-Value 0.366
Mean 2.4771StDev 0.9799Variance 0.9602Skewness 0.683738Kurtosis 0.408484N 36
Minimum 0.6522
Anderson-Darling Normality Test
95% Confidence I nterval for Mean
95% Confidence I nterval for Median
95% Confidence I nterval for StDev95% Confidence I ntervals
Summary for Ratio Total_O
San Diego, CAJanuary 18-19, 2011
Registration Management Committee (RMC)
28
Ratio Majors/Entries
On average, one Major is written every fifth audit – NOTE: Average down to .21 per audit for 2010 (1/5 audits)
2009Major Histogram
All CRB's
0
5
10
15
20
25
30
0.0 to <= 0.317 0.317 to <= 0.634 0.634 to <= 0.951 0.951 to <= 1.268
Class
# O
bs
erv
ati
on
s
2009Normal Distribution Mean = 0.2642Std Dev = 0.294KS Test p-value = .0663
2008Normal Distribution Mean = 0.2576Std Dev = 0.2863KS Test p-value = .0079
Totals
2010 – 1117 YTD
2009 -1135
2008 – 976
2007 - 870
San Diego, CAJanuary 18-19, 2011
Registration Management Committee (RMC)
29
OASIS Ratio of Majors
NOT Normal Distribution
1.251.000.750.500.250.00-0.25-0.50
99
95
90
80
70
605040
30
20
10
5
1
Ratio Majors_0
Pe
rce
nt
Mean 0.2106StDev 0.2694N 37RJ 0.828P-Value <0.010
Probability Plot of Ratio Majors_0Normal
1.20.90.60.30.0
Median
Mean
0.300.250.200.150.10
1st Quartile 0.06676
Median 0.117193rd Quartile 0.28934Maximum 1.20000
0.12075 0.30043
0.08414 0.17702
0.21912 0.35000
A-Squared 3.95P-Value < 0.005
Mean 0.21059StDev 0.26945Variance 0.07260
Skewness 2.50229Kurtosis 6.35899N 37
Minimum 0.00000
Anderson-Darling Normality Test
95% Confidence I nterval for Mean
95% Confidence I nterval for Median
95% Confidence I nterval for StDev95% Confidence I ntervals
Summary for Ratio Majors_0
Identify CB’s in the bottom 5% writing majors
San Diego, CAJanuary 18-19, 2011
Registration Management Committee (RMC)
30
OASIS Ratio of Majors - Modified
Normal Distribution after a transformation
210-1-2
99
95
90
80
70605040
30
20
10
5
1
Ratio Majors_0
Perc
ent
Goodness of Fit Test
AD = 0.303 P-Value = 0.556
Probability Plot for Ratio Majors_0
After Johnson transformation
Normal
San Diego, CAJanuary 18-19, 2011
Registration Management Committee (RMC)
31
Ratio Minors/Entries
On average, 2.8 Minors are written every audit, down to 2.47 minors per audit
2009Minor Histogram
All CRB's
0
2
4
6
8
10
12
0.786 to <=1.465
1.465 to <=2.144
2.144 to <=2.823
2.823 to <=3.502
3.502 to <=4.181
4.181 to <=4.859
4.859 to <=5.538
Class
# O
bs
erv
ati
on
s
2009Normal Distribution Mean = 2.839Std Dev = 1.1191KS Test p-value = .5280
2008Normal Distribution Mean = 2.8027Std Dev = 1.0927KS Test p-value = .5434
Totals
2010 – 15,814 YTD
2009 -19,414
2008 -17,393
2007 –14,087
San Diego, CAJanuary 18-19, 2011
Registration Management Committee (RMC)
32
OASIS Ratio of Minors
Normal Distribution
5.64.84.03.22.41.60.8
Median
Mean
2.82.62.42.22.01.8
1st Quartile 1.5947
Median 2.25583rd Quartile 2.9956Maximum 5.8333
2.0212 2.6932
1.7428 2.6773
0.8196 1.3092
A-Squared 0.54P-Value 0.158
Mean 2.3572StDev 1.0078Variance 1.0158
Skewness 1.09387Kurtosis 2.49466N 37
Minimum 0.5652
Anderson-Darling Normality Test
95% Confidence I nterval for Mean
95% Confidence I nterval for Median
95% Confidence I nterval for StDev95% Confidence I ntervals
Summary for Ratio Minors_O
543210
99
95
90
80
70
605040
30
20
10
5
1
Ratio Total_O
Pe
rce
nt
Mean 2.477StDev 0.9799N 36RJ 0.981P-Value >0.100
Probability Plot of Ratio Total_ONormal
Identify CB’s in the bottom 5% writing minors
San Diego, CAJanuary 18-19, 2011
Registration Management Committee (RMC)
33
AS9100 Nonconformities Quantities
major minor sum7.5 Production and Service Provision 145 2669 28144.2 Documentation Requirements 72 2088 21607.4 Purchasing 106 1864 19708.2 Monitoring and Measurement 163 1594 17578.5 Improvement 129 1106 12357.6 Control of Monitoring and Measuring Equipment 69 996 10658.3 Control of Nonconforming Product 57 675 7326.2 Human Resources 37 638 6755.6 Management Review 70 499 5697.2 Customer Related Processes 18 433 4514.1 General Requirements (Quality Management System) 18 271 2897.3 Design and Development 16 211 227
NO CHANGE
San Diego, CAJanuary 18-19, 2011
Registration Management Committee (RMC)
34
AS9100 Nonconformities Quantities (cont.)
major minor sum5.4 Planning (Management Responsibility) 15 182 1976.3 Infrastructure 9 153 1627.1 Planning of Product Realization 3 102 1058.4 Analysis of Data 7 86 934.3 Configuration Management 3 89 926.4 Work Environment 3 71 745.5 Responsibility, Authority and Communication 0 64 645.3 Quality Policy 2 49 515.1 Management Commitment 6 19 258.1 General (Measurement, Analysis and Improvement) 3 15 185.2 Customer Focus 3 13 166.1 Provision of Resources 2 9 11
NO CHANGE
San Diego, CAJanuary 18-19, 2011
Registration Management Committee (RMC)
35
Time – Line, Data Presentation
JAN MAR JUL SEP
First 6 Months
Summary
Previous Year
Summary
RULE VIOLATORS
San Diego, CAJanuary 18-19, 2011
Registration Management Committee (RMC)
36
0 50 100 150 200 250 300 350
6.1 Provision of Resources
5.2 Customer Focus
8.1 General (Measurement, Analysis and Improvement)
5.1 Management Commitment
5.3 Quality Policy
5.5 Responsibility, Authority and Communication
6.4 Work Environment
4.3 Configuration Management
8.4 Analysis of Data
7.1 Planning of Product Realization
6.3 Infrastructure
5.4 Planning (Management Responsibility)
7.3 Design and Development
4.1 General Requirements (Quality Management System)
Major Minor