AOAP eBallot - Sequence IVB is suitable for inclusion in ILSAC GF-6A and GF-6B The AOAP met on March 15, 2018 to review the “Sequence IVB Development” program. After the review a motion was made to Ballot the “Sequence IVB is suitable for inclusion in ILSAC GF-6A and GF-6B”. The motion detail is: Motion Sequence IVB meets the defined needs in the GF-6 Needs Statement for measuring low temperature engine wear performance of an engine oil and is suitable for inclusion in ILSAC GF-6A and GF-6B, realizing that the Sequence IV Surveillance Panel is evaluating Fe Content as an additional test parameter to complement the intake lifter average volume loss parameter. Motion by: Teri Kowalski, Toyota Seconded by: Ron Romano, Ford The AOAP Voted by Hand on accepting the Seq. IVB Motion. Hand Vote results: Affirmative = 6 Negative = 5 Abstain = 14 The Hand Vote results require that the Motion be resolved with a written eBallot. AOAP members should Vote on the motion “Sequence IVB is suitable for inclusion in ILSAC GF-6A and GF-6B” using the eBallot at http://ballots.api.org/login.aspx . The Sequence IVB GF-6 Motion and the supporting documentation are available the eBallot Website. This eBallot will close on Friday March 30, 2018. All Negative Votes must include comments which: a) Describes the section to which the negative ballot pertains b) Gives substantive reason(s) for negative vote. c) Proposes wording or action to resolve negative vote. All Abstentions/Waves to the AOAP Vote must include comments which: a) Gives substantive reason(s) for Abstain vote. b) Proposes wording or action to resolve negative vote. Any questions, please contact API.
Transcript
AOAP eBallot - Sequence IVB is suitable for inclusion in ILSAC GF-6A and GF-6B
The AOAP met on March 15, 2018 to review the “Sequence IVB Development” program. After the review a motion was made to Ballot the “Sequence IVB is suitable for inclusion in ILSAC GF-6A and GF-6B”.
The motion detail is:
Motion Sequence IVB meets the defined needs in the GF-6 Needs Statement for measuring low temperature engine wear performance of an engine oil and is suitable for inclusion in ILSAC GF-6A and GF-6B, realizing that the Sequence IV Surveillance Panel is evaluating Fe Content as an additional test parameter to complement the intake lifter average volume loss parameter.
Motion by: Teri Kowalski, Toyota
Seconded by: Ron Romano, Ford
The AOAP Voted by Hand on accepting the Seq. IVB Motion.
The Hand Vote results require that the Motion be resolved with a written eBallot.
AOAP members should Vote on the motion “Sequence IVB is suitable for inclusion in ILSAC GF-6A and GF-6B” using the eBallot at http://ballots.api.org/login.aspx .
The Sequence IVB GF-6 Motion and the supporting documentation are available the eBallot Website.
This eBallot will close on Friday March 30, 2018.
All Negative Votes must include comments which:
a) Describes the section to which the negative ballot pertainsb) Gives substantive reason(s) for negative vote.c) Proposes wording or action to resolve negative vote.
All Abstentions/Waves to the AOAP Vote must include comments which: a) Gives substantive reason(s) for Abstain vote.b) Proposes wording or action to resolve negative vote.
Any questions, please contact API.
AOAP Motion
• Sequence IVB meets the defined needs in the GF-6 NeedsStatement for measuring low temperature engine wearperformance of an engine oil and is suitable for inclusion inILSAC GF-6A and GF-6B, realizing that the Sequence IVSurveillance Panel is evaluating Fe Content as an additional testparameter to complement the intake lifter average volume lossparameter.
Motion by Teri KowalskiSeconded by Ron Romano
15
PCEOCP / AOAPMarch 15, 2018
Teri KowalskiWilliam A Buscher III
1
Summary of Activity Since 12/7/17
• Completed Precision Matrix 2 testing at Intertek, SwRI, Lubrizol and ExxonMobil
• Completed prove-out testing at Afton• To-date a total of 45 prove-out and precision matrix tests have
been completed• Completed statistical analysis of Precision Matrix 2
– Analysis performed for two data sets• N = 28, includes independent and dependent labs• N = 21, includes independent labs (official precision matrix design)
– Data supports the use of Sqrt(AVLI) transformation– Significant oil difference: 1012 < 300– Lab differences are statistically different
• Surveillance panel addressing
– Stand within lab differences are not significantly different– Reference oil targets established 2
Prove-out and Precision Matrix 2 Results
3
IAR Stand 3 IAR Stand 4 SwRI Stand 1 SwRI Stand 2 SwRI Stand 3 Lubrizol ExxonMobil Afton
1 1.79 N/A 1.01 N/A 0.89 1.10 1.53 2.40
2 1.32 2.26 1.16 N/A 1.46 1.31 1.04
3 1.06 1.15 1.50
Run OrderRequired Supplemental
PROVE-OUT TESTING
IAR Stand 1 IAR Stand 2 IAR Stand 3 SwRI Stand 1 SwRI Stand 2 Lubrizol ExxonMobil Afton
• Sequence IV surveillance panel met on 1/11/18, 1/25/18, 3/1/18 and 3/7/18– Reviewed additional prove-out testing
• Operational data and test results
– Reviewed Precision Matrix 2 testing• Extensive operational data analysis and review including:
– 1-hour operational data plots from start, mid and end of test– 200-hour operational data plots– Statistical analysis of operational ramp data– Statistical analysis of operational data correlation to Sqrt(AVLI)
• N = 28 and N = 21 Precision Matrix 2 statistical analyses review
– Approved statistical analysis of Precision Matrix 2– Reviewed results from potential high wear candidate oils– Voted that the test is ready for inclusion into GF-6 and to become an
ASTM procedure– Reviewing LTMS examples– Next meeting planned for week of March 18th
6
Sequence IV Surveillance Panel Motion
• The Sequence IV Surveillance Panel, having secured hardware supply, test fuel and reference oils for a test procedure that measures the performance of passenger car motor oil for low temperature engine wear, recommends to the Passenger Car Engine Oil Classification Panel, the Auto Oil Advisory Panel and the American Chemistry Council that the Sequence IVB test is ready for inclusion in ILSAC GF-6 and that the Sequence IVB procedure be published as an ASTM method. Realizing that the test parameters (AVLI and Fe content) need to be finalized and the LTMS still needs to be developed.
Teri Kowalski / Ron Romano / Passed 11 – 2 – 8
7
Summary of Activity Since 12/7/17
• Toyota solicited oil suppliers for Sequence IVB results from potential high wear oils– Data from 3 oils from 3 suppliers was presented to Toyota and the
Sequence IV surveillance panel– 1 of the 3 oils produced high valve-train wear and very high Fe content
at EOT, indicating high overall engine wear• Much higher Fe versus valve-train wear than the typical valve-train wear to Fe
correlation
– 1 of the 3 oils produced moderate valve-train wear and high Fe content at EOT, indicating high overall engine wear
• Much higher Fe versus valve-train wear than the typical valve-train wear to Fe correlation
– 1 of the 3 oils produced low valve-train wear and low Fe content at EOT, but is suspect of having formulation components that relaxed the oil degradation mechanism of the Sequence IVB test
• Surveillance panel action item for supplier to investigate and report back
8
Summary of Activity Since 12/7/17
• Toyota solicited oil suppliers for Sequence IVB results from potential high wear oils– Conclusions:
• Sequence IVB evaluates more than just the performance of a passenger car motor oil for low temperature valve-train wear, but evaluates the performance of a passenger car motor oil for low temperature engine wear
• Fe content is an indicator of engine wear, and is important, in addition to average intake volume loss
• Sequence IVB responds to a variety of passenger car motor oil formulation components
9
Average Intake Volume Loss by Oil (N = 30)
10
X invalid
Summary of Activity Since 12/7/17
• LTMS examples established and distributed to the Sequence IV surveillance panel for review and approval, with the following suggestions:– Stand based system– Calibration period: fifteen full length non-reference tests or 6 months– Reference oils and assignment: 300 (40%), 1012 (40%) and 1011 (20%)– A minimum of two reference tests will be required for each new stand– Adopt the transform Sqrt(AVLI) for LTMS and severity adjustment
calculations– Select reference oil targets from presented N = 28 or N = 21 models– Utilize limits on Zi (EWMA of severity), ei (prediction error), and the
excessive influence calculation to determine acceptance and calculate severity adjustments
– The TMC will plot industry Zi charts to identify potential shifts in industry wide performance
11
Precision Matrix 2 LTMS Example N = 21
12
Precision Matrix 2 LTMS Example N = 21
13
PCEOCP Motion
• Sequence IVB is suitable for measuring low temperature engine wear performance of an engine oil and is recommended for adoption as an ASTM procedure, realizing that the Sequence IV Surveillance Panel is evaluating Fe Content as an additional test parameter to complement the intake lifter average volume loss parameter.
Motion by Teri KowalskiSecond by Ron Romano
Discussion
14
AOAP Motion
• Sequence IVB meets the defined needs in the GF-6 Needs Statement for measuring low temperature engine wear performance of an engine oil and is suitable for inclusion in ILSAC GF-6A and GF-6B, realizing that the Sequence IV Surveillance Panel is evaluating Fe Content as an additional test parameter to complement the intake lifter average volume loss parameter.
Motion by Teri KowalskiSeconded by Ron Romano
15
AOAP Motion 2
• Include Seq. IVB in GF-6A and GF-6B for the purpose of demonstration of Low Temperature Engine Wear Phenomena using rate and report procedures.
Motion by Matthew AnsariSeconded by TABLED w/o Second
– This analysis includes the results of 28 valid precision matrix tests– Data supports the use of Sqrt(AVLI) transformation – Significant oil differences: 1012 < 300– Lab differences are statistically significant (A < B1)– Stand within Lab differences are not statistically significant– Estimated within a stand test precision (r; ASTM repeatability)
• Sqrt(AVLI) = 0.4657– Estimated test precision across labs and stands (R; ASTM reproducibility)
• Sqrt(AVLI) = 0.5552– Oil means and standard deviations
• The two high results on Oil 300 at stands B1-2 and B1-3 have large influence on discrimination between oils 300 and 1012. Without these two tests, differences between oils are not statistically significant.
• Discrimination is not consistent among the stands.– Labs F and G may not discriminate oils– Stands rank oils differently– This could be an issue if the same phenomenon is observed in candidate
oils• Test precision is large compared to the observed range of measurements; lab
differences are larger than oil differences; the high and low oils diff by 1.4 standard deviations (lowest of any GF6 test).– The resulting LTMS would likely allow calibration of stands that don’t
discriminate oils– Discriminating future oils in the test will be difficult; especially with only
one test result
PM Analysis Comments - 1
21
• Statisticians chose to weight targets by lab (25% per lab) rather than by stand (approx. 14% per stand). The effect is that the average of a lab with 3 stands and the average of a lab with 1 stand will have the same 25% weighting on the targets. This was seen by stats group to better represent industry-wide performance, align with past analyses’ methodology, and does not affect any results other than the targets. Stand weighted targets could be pursued if the panel desires.
• Some belief amongst some stat group members that transforming individual lifter results before averaging may be more appropriate than transforming the average. Since the benefit of doing this new approach was minimal and time was short, this analysis is included in the appendix only.
• AVLI in the LTMS file is sometimes off in the hundredths place from the calculated average of the eight lifters shown in the same file. Impact is negligible, but the source of the AVLI in the LTMS file should be made clear.
PM Analysis Comments - 2
22
• Based on analysis conducted, there is no additional benefit in using parameters other than AVLI
• Additional tests could help better understand discrimination and precision of the test.
• Statistical analyses have not yet been completed to assess the impact of operational differences on test severity. The outcome of such analyses and discussions could ultimately affect oil targets. Given the differences noted in the surveillance panel call on January 11th, the panel may find it helpful to review the full datafiles for all tests.
• A review of individual lifter measurements suggests some merit to the incorporation of an outlier screening methodology– An initial review of the impact of outlier screening indicates minimal improvement
in oil discrimination and precision• It is unknown whether or not the number of outliers for candidate oil tests are
more likely to occur as compared to reference oil tests. (Greater number of outliers in candidate oils would make a stronger case for outlier screening.)
• Lifter bias is observed and can be taken into account in outlier screening methods
– Some methodologies investigated included:• Removal of the max and min lifter result of both non-transformed and mean-
centered lifter data• Weighted average with higher weights for lifters that differ• Similar approach to what is done for T12 and C13 for performance properties
with lifter bias• Evaluating several outlier screening methods listed in E178.
• Outlier screening can be pursued further if the surveillance panel deems it appropriate; the final methodology will likely impact oil targets that are established using non-screened lifter measurements 23
Additional Comments - 1
• An initial review of the impact of lifter weighting indicates minimal improvement in oil discrimination and precision
• Lifter weighting can be pursued further if an engineering reason exists for the differences by position
24
Additional Comments - 2
Data Utilized
25
– Precision Matrix Data: • 4 Labs {A, B1, F and G}• 3 Reference Oils {300, 1012, and 1011}• 7 Stands {A-1, A-2, B1-1, B1-2, B1-3, F-1 and G-1}• Total number of tests = 28
– Precision Matrix Data Table from Rich Grundza’s 20180115 IVB Matrix update.
Parameter Abbreviation
26
– AVLI - Average volume loss, Intake– AMLI - Average mass loss, Intake– AVLOSEXK - Average volume loss, Exhaust– AMLOSEX - Average mass loss, Exhaust– SumVLIE - Average volume loss, Intake +
Exhaust– SumMLIE - Average mass loss, Intake + Exhaust – FEWMEOT – Fe-Wear Metals at end of test
Data Calculation
27
• AMLI and AMLOSEX – For Lab G data, multiplied individual lifter mass loss by 1000 and took the average of 8 lifters – Remove test 130943-IVB’s BL2EXHML = 655.1 in
calculating average which results to AMLOSEX=9.6 • AVLOSEXK
– Remove Lab A test 130948-IVB’s BL1EXKVL=-0.2 in calculating average which results to AVLOSEXK=0.85
– Lab G did not measure AVLOSEXK for test 130944-IVB • SumVLIE = AVLI + AVLOSEXK• SumMLIE = AMLI + AMLOSEX
Summary of Model Results
28
Most parameters except AMLI show that lab difference is greater than oil difference. The Volume Loss parameters showed no significant difference between stands within the lab.
Note: n-size for these models is 28 except for SumVLIE and AVLOSEXK which has 27
The table below compares the numbers of standard deviations of separation between the highest and lowest reference oil across GF-6 test types. The median is approx. 3.3 and the mean (without PHOS) is 3.4.
*1: Oil 220 not used as a reference oil. Including this oil would yield approx. 12 SDs of separation between 220 and 222.*2: 271 vs. 1011
AVLIAverage volume loss, Intake
30
Average Intake Volume Loss by Oil
31
• The below plot summarizes the AVLI test result data by reference oil.
Average Intake Volume Loss by Stand
32
• It appears that oil discrimination is not consistent among the stands; Labs F and G may not discriminate oils; Stands rank oils differently
Average Intake Volume Loss by Lab
33
• Below plot summarizes the AVLI test result data by test Lab and reference oil
Sqrt(AVLI) ANOVA Full Model
34
Statistically significant differences:• Oil• LabNot significantly different:• Stands within Labs
Sqrt(AVLI) Oil Differences
35
• Model is Sqrt(AVLI) ~ Oil, Lab, Stand(Lab)• Oils significantly differ
• Oil 300 is statistically significantlydifferent than oil 1012
• Oil 1011 is not statistically significantlydifferent than oils 300 and 1012
• Plot shows Sqrt(AVLI) LSMeans by Oil,with 95% confidence intervals
• Model is Sqrt(AVLI) ~ Oil, Lab, Stand(Lab)• Plot below of Sqrt(AVLI) LSMeans by Lab, with 95% confidence intervals• Lab A is statistically significantly different than Lab B1.
LSMeans by Lab
LSMeans Differences Between Labs
LabSqrt(AVLI)
LSMeanAVLI
LSMean A 1.1298 1.28 B1 1.3882 1.93 F 1.1789 1.39 G 1.3713 1.88
Lab1 Lab2
Sqrt(AVLI) LSMean
Difference p-Value B1 A 0.2584 0.01 G A 0.2415 0.12 B1 F 0.2093 0.27 G F 0.1924 0.49 F A 0.0491 0.97
B1 G 0.0169 1
Sqrt(AVLI) Stand within Lab Differences
37
• Model is Sqrt(AVLI) ~ Oil, Lab, Stand(Lab)• Plot below of Sqrt(AVLI) LSMeans by Stand, with 95% confidence
intervals• Stands within labs are not statistically significantly different from each
– This analysis includes the results of 21 valid precision matrix tests from the independent labs
– Data supports the use of Sqrt(AVLI) transformation – Significant oil differences: 1012 < 300– Lab differences are statistically significant (A < B1)– Stand within Lab differences are not statistically significant– Estimated within a stand test precision (r; ASTM repeatability)
• Sqrt(AVLI) = 0.4593– Estimated test precision across labs and stands (R; ASTM reproducibility)
• Sqrt(AVLI) = 0.5771– Oil means and standard deviations
• The two high results on Oil 300 at stands B1-2 and B1-3 have large influence on discrimination between oils 300 and 1012. Without these two tests, differences between oils are not statistically significant.
• Discrimination is not consistent among the stands.– Stands rank oils differently– This could be an issue if the same phenomenon is observed in candidate
oils
• Test precision is large compared to the observed range of measurements; the high and low oils differ by 1.9 standard deviations (lowest of any GF6 test).– Discriminating future oils in the test will be difficult; especially with only
one test result
Data Utilized
45
– Precision Matrix Data: • 2 Labs {A, B1}, independent labs only• 3 Reference Oils {300, 1012, and 1011}• 5 Stands {A-1, A-2, B1-1, B1-2, B1-3}• Total number of tests = 21
– Precision Matrix Data Table from Rich Grundza’s 20180115 IVB Matrix update.
Reference Oil Discrimination Comparison
46
Test Parameter Oil 1 Oil 2 Range Test SDs of Separation
The table below compares the numbers of standard deviations of separation between the highest and lowest reference oil across GF-6 test types. The median is approx. 3.3 and the mean (without PHOS) is 3.4.
*1: Oil 220 not used as a reference oil. Including this oil would yield approx. 12 SDs of separation between 220 and 222.*2: 271 vs. 1011
ANALYSIS OF SQRT(AVLI)Average volume loss, Intake
47
Average Intake Volume Loss by Oil
48
• The below plot summarizes the AVLI test result data by reference oil.
Average Intake Volume Loss by Stand
49
• It appears that oil discrimination is not consistent among the stands; Stands rank oils differently
Average Intake Volume Loss by Lab
50
• Below plot summarizes the AVLI test result data by test Lab and reference oil
Sqrt(AVLI) ANOVA Full Model
51
Statistically significant differences:• Oil• LabNot significantly different:• Stands within Labs
Sqrt(AVLI) Oil Differences
52
• Model is Sqrt(AVLI) ~ Oil, Lab, Stand(Lab)• Oils significantly differ
• Oil 300 is statistically significantlydifferent than oil 1012
• Oil 1011 is not statistically significantlydifferent than oils 300 and 1012
• Plot shows Sqrt(AVLI) LSMeans by Oil,with 95% confidence intervals
• Model is Sqrt(AVLI) ~ Oil, Lab, Stand(Lab)• Plot below of Sqrt(AVLI) LSMeans by Lab, with 95% confidence intervals• Lab A is statistically significantly different than Lab B1.
LSMeans by Lab
LSMeans Difference Between Labs
LevelSqrt (AVLI)
LSMeanAVLI
LSMean A 1.1307 1.28 B1 1.3882 1.93
Lab1 Lab2
Sqrt(AVLI) LSMean
Difference p-Value B1 A 0.2575 0
Sqrt(AVLI) Stand within Lab Differences
54
• Model is Sqrt(AVLI) ~ Oil, Lab, Stand(Lab)• Plot below of Sqrt(AVLI) LSMeans by Stand, with 95% confidence
intervals• Stands within labs are not statistically significantly different from each
