Proficiency Testing

Paul Armishaw

Topics

• The NMI PT program• Traceability and PT• Measurement uncertainty and PT• Case study – folic acid in flour

Proficiency Testing

• Proficiency testing is:– “Evaluation of participant performance against

pre-established criteria by means of interlaboratory comparisons.” ISO 17043

• NMI is an accredited PT provider– by NATA– to ISO 17043

The NMI PT Program• Trace metals

– soil, water, pharmaceuticals, foodstuffs

• Hydrocarbon pollutants– water and soil

• Forensic drugs– heroin, cocaine, amphetamine and amphetamine-like substances– clandestine laboratories

• Allergens• Folic acid

– bread making flour

• Pesticide residues– soil– water– fruit and vegetable

• Dimethylsulfidepropionate in seawater– a natural anti-greenhouse compound

• Emerging pollutants (PFOS and PFOA)

The Role of NMI PT

NMI PT

Policing Education

“Nothing is gained from PT participation unless the information received is directed to improvement in the laboratory.”

Mechanics of a PT Round

Prepare sample

Homogeneity Testing

Distribute

Participants perform analysis

Spike with analyteHomogenize, package

Reference method egIsotope Dilution Mass Spectrometry (IDMS)Or other means

Stability, integritycustoms and quarantineNormal test method

Return results 4 weeks

PT Report Statistical analysisPerformance assessmentMethod information

QualityImprovement

Technical seminars

Reference methods

Reference materials

Training

Traceability• NMI provide traceable assigned values of

selected analytes.– folic acid in flour– pesticide in fruit and veg– metals in water

• Assigned value independent of the participants’ results

• Traceable to SI• Useful where there is

– biased methodology– no consensus

Chlorpyrifos in rockmelon• No consensus amongst the participant labs• Reference value traceable to the SI isotope dilution mass spectrometry• Assigned value result of reference measurements

0.00

0.01

0.02

0.03

0.04

0.05

0.06

21 5 19 3 16 13 14 2 20 23 25 1 4 9 18 15 24 17 10 11 12 22 6 7 26 Md Rv

Laboratory

Chl

orpy

rifos

(mg/

kg)

Assigned value 0.034 ± 0.002 mg/kg

TraceabilityWhat are we measuring?

• Chlorpyrifos in rockmelon– analyte is clearly defined small molecule

– result is independent of the measurement method

• Gluten in food– gluten is group of proteins MW 30,000 - 107

– measured using proprietary kits

Gluten in Custard PowderLab Code Target Measurand of Kit

1 Gliadin, glutenin2 Gliadin3 Gliadin4 omega-gliadin5 Gliadin6 prolamins from wheat, rye, barley7 Gliadin8 Glutenin and Gliadin9 Gliadin (also glutenin)10 Gliadin11 Gliadin

Duplicate samplesGluten in Custard Powder

0

5

10

15

20

25

30

35

40

2 2 11 11 8 8 9 9 3 3 1 1 6 6 7 7 4 4 5 5 10 10

Lab code

Glu

ten

mg/

kg

Kit A

Kit B

Kit C

Uncertainty

• A requirement of ISO 17025 since 2000• ISO AS 15189 - 2013

– ‘shall determine measurement uncertainty’• Well established in accredited chemical

testing laboratories• NMI asks that labs report uncertainties• MU assessed using En-score• MU estimates have improved as labs gain

experience

MU: State of Practice, 2004Cu in Wheat Flour

0

2

4

6

9 9 8 8 7 7 3 3 10 10 4 4 5 5 11 11 1 1 2 2 6 6 Md Hom RV

Laboratory

Res

ult (

mg/

kg)

Assigned Value = 3.87 ± 0.17 mg/kg

AQA 04-04

Much Guidance Published

MU: State of Practice, 2011Cu in Wheat Flour

0

1

2

3

4

5

6

7

8

1 10 9 2 3 7 6 8 11 5 4 Md R.A. H.V.

Laboratory

Res

ults

(mg/

kg)

Assigned Value = 3.94 ± 0.12 mg/kg

AQA 11-14

PT: a Reality Check On MU

nE = En-score = individual laboratory result

U = expanded uncertainty of the individual laboratory result = assigned value

XU = expanded uncertainty of the assigned value.

• |En| ≤1 Satisfactory• |En| > 1 Unsatisfactory

22

)(

X

nUUXE

Does my results agree with the assigned value, taking account of their uncertainties?

The En-score expresses mathematically what can be seen on the results chart.

0

1

2

3

4

5

6

7

8

1 10 9 2 3 7 6 8 11 5 4 Md R.A. H.V.

Laboratory

Res

ults

(mg/

kg)

Assigned Value = 3.94 ± 0.12 mg/kg

En = -1.3 En = -0.5

AQA 11-14

Same Cu in Wheat Flour

Folic Acid PT Program Mandatory fortification of bread making flour introduced in Australia in 2009

“no less than 2 mg/kg and no more than 3 mg/kg”

Advisory GroupRepresentatives from– regulatory body

• Food Standards Australia New Zealand

– flour milling industry– testing laboratories

Aims of PT Program– assist laboratories with method implementation and accreditation to ISO 17025– asses the accuracy of participant laboratories in the identification and

measurement of folic acid in flour – assess the comparability of results– assess the fitness for purpose

Sample Preparation – Stage 1White flour

(200 g) Folic acid

standard (5 g)

5 g Folic acid standard + 45 g of white flour

Ready to use for fortifying test samples (2500 mg/kg)

STEP 1:

STEP 2: 5 g of the above mixture + 195 g of white flour

Mixed thoroughly using a mini drum

hoop mixer

Sample Preparation – Stage 2White flour or grain

bread-mix Concentrated pre-mix

Sterilise (-80C overnight)

Mixed in incremental steps using a mini drum hoop mixer

Dispense 20 g into polypropylene jars

Results Trial Round Unfortified flour

• Significant levels of folic acid reported• These are false positive results

0.02 0.08 0.150.345

0.53 0.59 0.7

1.09

0.438

0.0

0.5

1.0

1.5

5 3 7 1 4 8 11 2 6 Md

Laboratory

Folic

Aci

d (m

g/kg

)

Results Trial Round Duplicate pair fortified at 2.5 mg/kg

• Within laboratory variability good• Between laboratory variability poor

0

1

2

3

4

5

1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 11 11 M S

Laboratory

Folic

Aci

d (m

g/kg

)

Results vs Method

No obvious trends with the method

HPLC Micro Assay Biosensor Not disclosed

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0Fo

lic A

cid

(mg/

kg) Spike level

Participants’ Teleconference

Organised by NMI to discuss results of trial round• 7 laboratories participated

Technical focus• why are the results so poor• what are the critical factors to achieve good results

Outcomes• suitable internal standards for HPLC• native folates should not be an issue in wheat flour• possibility of folic acid forming metal complexes during HPLC

analysis• need for suitable reference materials• design of future PT rounds• calibration standards

Reference value The primary method was developed by NMI

Reference values were obtained using exact matching Isotope Dilution with liquid chromatography tandem mass spectrometry. Isotopically labelled 13C5-folic acid used as internal standard Three characteristic transition ions monitored

CalibrationBlend

SampleBlend

Third round of PT

Results much better after three rounds of PT But still not perfect Folic acid in flour CRM available from NMI

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

1 1 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 11 11 12 12 RA S RV

Laboratory

Folic

Aci

d m

g/kg

Assigned Value 2.22 ± 0.07 mg/kg

RA - Robust AverageS - Spike RV - Reference Value

Upper compliance limit

Lower compliance limit

NMI PT and Laboratory Performance

• Not blind – laboratories know it is a PT sample

• Limited range of target analytes– list provided to participants

• Can compare with a blank– laboratories get a portion of the un-spiked matrix

• Long turn around time– four weeks

• So what we see is the laboratory’s best attempt• PT is an important part of a quality system, but:

– PT is just a snapshot– anyone can make a mistake

Types of errors• Two-thirds of errors are technical

– extraction method– instrumental technique– dilution factors– interference correction– calibration– etc

• One-third are blunders– wrong units– transcription error– calculation error– sample mix-up

Acknowledgements

• NATA for the invitation

• NMI colleagues– Geoff Morschel– Luminita Antin– Raluca Iavetz

• Participants in the NMI PT program

• More information– www.measurement.gov.au– proficiency@measurement.gov.au

National Measurement Institute105 Delhi RoadNorth Ryde, NSW, Australia

Phone: + 61 2 9449 0111

Email: proficiency@measurement.gov.au