Post on 24-Feb-2016
description
transcript
How proficient are you?
Sylvia Janetzki, M.D.ZellNet Consulting, Inc.
Cancer Immunotherapy Consortiumsylvia@zellnet.com
How proficient are youreally?
Sylvia Janetzki, M.D.ZellNet Consulting, Inc
Cancer Immunotherapy Consortiumsylvia@zellnet.com
Proficiency:
Mastery of a specific skill demonstrated by consistently superior performance, measured against established or popular standards.
Business dictionary
External QA in immune monitoring
CLSI (former NCCLS) document I/LA26-A, Vol. 24, No. 29 “Performance of Single Cell Immune response assays; approved guidelines”
“Each new test must be validated before beingimplemented for patient testing… Validation of a test procedure includes the external quality assessment through a proficiency testing program.” (p47)
What is a proficiency panel?
• Same samples sent to each lab
• Each lab tests samples under own SOP
• Each lab reports measured values back
• Those values are compared to
A predefined reference value, or
The results reported by all panelists
• Labs receive a performance report
Every immune monitor”er” can get proficiency tested and should get proficiency tested because
1. participation in proficiency testing is a valuable continuous improvement tool;
2. it gives direct feedback about your performance and can provide reassurance;
3. it builds credibility and confidence with sponsoring and regulatory agencies;
4. it helps accelerate vaccine development through accurate immune monitoring;
5. it is an ethical responsibility to the clinical trial volunteer to create accurate and reliable data.
100
200
300
Min=8
Max=281
Median=109
CV=55
Mea
n sp
ot n
umbe
r per
wel
lElispot
1
2
3
Mea
n %
Cyt
okin
e+ C
D4+
lym
phoc
ytes
Max=3.29
Min=0.16
Median=1.93
ICS
CV=47
Mea
n %
CD
8+ M
ultim
er B
indi
ng
0.30
0.60
1.5
Min=0
Max=1.53
Median=0.16
CV=135
HLA-peptide multimer staining
37 labs, mean of6 replicates per lab
27 labs, mean of3 replicates per lab
28 labs, mean of3 replicates per lab
“moderate” response “strong” response “low” response
0.15
0.45
1.35
Because:
Janetzki et al. Immunity 2009
My assay is the best
I only believe my own
data
I am well Known in my field.Hence – myassays mustbe good.
There is a lack of a Gold Standard (test) for assays assessing responses on single cell level
And because:
How do we know what works best?
There is no gold standard for most commonly used immune monitoring assays.
Crucial protocol variables can be identified that influence assay outcome in either direction.
The check-up on performance can be achieved via inter-laboratory testing projects = proficiency panels*.
* The larger the panel, the more powerful the results (panel design essential)
Harmonization Guidelines
Phases of assay “evolution”
Development
Qualification
Optimization
Standardization
Validation
ImplementationRe-validation
External QA=
Proficiency Panel
Harmonization guidelines
“Gold Standard”
2005: 1st Elispot panel (36 labs)2006: 2nd Elispot panel (29 labs)2007: 3rd Elispot panel (35 labs) 1st Multimer panel (29 labs) 1st ICS panel (28 labs) 1st CFSE panel (21 labs)2009: 4th Elispot panel (41 labs)
2nd Multimer panel (20 labs) 2nd ICS panel (31 labs)2010: ICS Gating panel (110 labs)2011-12: 5th Elispot panel (40 labs) 3rd Multimer panel (35 labs) 1st – 3rd Luminex panel (26 labs)2013: 1st Elispot Plate Reading panel (86 labs)
2nd ICS Gating Panel
20 countries
The CIC Proficiency Panel Program
• The first panel was run under highly standardized conditions with standardized reagents (BD lyoplates).
• A high degree of variability and failure of many labs to detect responses was observed.
• Most of the variation was due to:– Number of cells counted (labs that acquired >100,000
cells had better results)– Instrument setup and compensation strategies (many
labs did not accurately compensate their samples)– Gating strategies
• 1st ICS Panel Findings: 28 labs participated
1st ICS Panel: Overall Variability Example
This was confirmed by other independent panels:
CIP/CIMT 2011 (Welters et al. 2011) NIAID 2010 (Jaimes et al. 2010)
Overview of ICS Gating Panel Design
• All participants analyzed the same set of nine FCS files. • No experiments were performed. • Setting up the study this way allows for quantification
of the variability caused by gating since all other experimental sources of variation are removed.
• The panel had two phases:• Phase I: Labs evaluated the FCS files using their
lab-specific protocol/gating strategy.• Phase II: Labs re-analyzed the same FCS files using
a gating strategy that was drafted by consensus from the organizers of the panel.
Lisa McNeil& ICSSteeringCommittee
FCS Files
• Nine FCS files (all labs received the SAME files)
• 3 donors (Donor 1, 2 and 3) that have been stimulated with CEF, CMV peptide pool and media (unstimulated).
• The cells have been stained with 5 antibodies:– Dump channel (CD14 and CD19 Pacific Blue)– CD3-FITC– CD4-PE-Cy7– CD8-Alexa 700 – IL-2/IFN-APC
• The participants were requested to evaluate the % of CD4+ cytokine+ cells and CD8+ cytokine+ cells and to provide gating
strategy and answer a survey.
• CIC/CRI – Jianda Yuan, Leah Price (Statistician), Sylvia Janetzki (Coordinator)
• CHAVI /EQAPOL- Janet Staats (Reviewer)• CVC/CRI - Kunle Odunsi• CIMT - Cedrik Britten (Reviewer)• DAIDS - Maria Jaimes (Reviewer)• FOCIS - Holden Maecker• IAVI - Dilbinder Gill• IDS - Ivana Durinovic-Bello, Jerill Thorpe (Reviewer)• iSBTc/SITC - Lisa Butterfield• ITN - Jason Liu
This was a huge effort – but well worth it
110 labs participated
Summary of Results
• 110 Labs participated and there were 110 different approaches to gating.
• Variability was dramatically reduced when all labs used the same gating strategy (Phase II).
• Proximity of the cytokine gate to the negative population impacted true positive and false positive response detection.
Variability is decreased if all labs use the same gating strategy.
Lab
CEF
Unstimulated
CMV pp65
CEF
Unstimulated
CMV pp65
Donor 1, CD8
Consensus
CEF
Unstimulated
CMV pp65
Lab
Consensus
Placement of Cytokine Gate
APC-A
Cou
nt
-102 102 103 104 1050
284
568
851
1135
M1
Proximity Adequate Too far HistogramToo close
CD4 Cytokine Gate: Gate
Proximity (% of labs)
5 or 6 responses
AND No false positives
(n=37)
5 or 6 responses
AND False
positive (n=44)
< 5 responses
(n=28)
< 4 responses
(n=13)
Adequate Proximity 54% 30% 32% 23%
Too Far 8% 64% 7% 0%
Too Close 38% 2% 57% 69%
21
CD4+ CD8+ T cells: Double Positive cells
CD4+ CD8+ DP cellsare excluded from the CD4 and CD8 gates
CD4+ CD8+ DP cells are includedin both CD4 and CD8 gates
DP Gate (% of labs)
5 or 6 responses
AND No false positives
(n=37)
5 or 6 responses
AND False
positive (n=44)
< 5 responses
(n=28)
< 4 responses
(n=13)
% of labs that excluded DP
cells 65% 73% 46% 62%
22
Lymphocyte Gate
Lymphocyte Gate (% of labs)
5 or 6 responses
AND No false positives
(n=37)
5 or 6 responses
AND False
positive (n=44)
< 5 responses
(n=28)
< 4 responses
(n=13)
Included all Lymphocytes, No debris or
RBC 49% 43% 43% 23%
Excluded Lymphocytes 19% 30% 14% 31%
Too Large 8% 20% 21% 23%
Included RBC and debris 19% 5% 18% 15%
Inclusion of Dim Cells
Dim cells (% of labs)
5 or 6 responses
AND No false positives
(n=37)
5 or 6 responses
AND False
positive (n=44)
< 5 responses
(n=28)
< 4 responses
(n=13)
Included CD3 Dim cells 46% 55% 25% 23%
Included CD4 Dim cells 46% 50% 43% 31%
Included CD8 Dim cells 46% 59% 46% 31%
Examples of Dims not included in the CD4/CD8 Gate
Uniformity of Gates
0 102 103 104 105
0
102
103
104
105
Donor 2_DMSO_D05.fcsÉCD8+CD4-
<Alexa Fluor 700-A>: CD8
<AP
C-A
>: IF
N-g
/IL-2
0.024
0 102 103 104 105
0
102
103
104
105
Donor 2_CEF_D03.fcsÉCD8+CD4-
<Alexa Fluor 700-A>: CD8
<AP
C-A
>: IF
N-g
/IL-2
0.67Changing gates within a donor
Uniform gates (% of labs)
5 or 6 responses
ANDNo false positives
(n=37)
5 or 6 responses
ANDFalse
positive(n=44)
< 5 responses
(n=28)
< 4 responses
(n=13)
Did not use uniform gates within a donor
3% 16% 11% 0%
Sometimes or never used
uniform gates across donors
17% 39% 18% 15%
Biexponential Scaling
CD8 gate not drawn all the
way to the axis, missing ½ of the
cells!
Biexponential scaling not correctly applied.
UngatedDonor 1_CEF_B03.fcsEvent Count: 169945
0 102 103 104 105<Pacific Blue-A>: Dump gate
0
50K
100K
150K
200K
250K
SSC-
A
85.9
dumpFirstDonor 1_CEF_B03.fcsEvent Count: 145945
0 50K 100K 150K 200K 250KFSC-A
0
50K
100K
150K
200K
250K
SSC
-A
66.2
lymphocytesDonor 1_CEF_B03.fcsEvent Count: 96631
0 102 103 104 105<PE-Cy7-A>: CD4
0102
103
104
105
<FIT
C-A
>: C
D3
46.7
CD3CD4Donor 1_CEF_B03.fcsEvent Count: 45137
0 102 103 104 105<PE-Cy7-A>: CD4
0102
103
104
105
<APC
-A>:
IFN-
g/IL
-2 CD3CD8Donor 1_CEF_B03.fcsEvent Count: 35708
0 102 103 104 105<Alexa Fluor 700-A>: CD8
0102
103
104
105
<APC
-A>:
IFN
-g/IL
-2
0 102 103 104 105<Alexa Fluor 700-A>: CD8
0102
103
104
105
<FIT
C-A
>: C
D3
37
lymphocytesDonor 1_CEF_B03.fcsEvent Count: 96631
0 102 103 104 105
0
102
103
104
105
Donor 1_DMSO ii_B06.fcsÉPacific Blue- CD3+
<PE-Cy7-A>: CD4
<Ale
xa F
luor
700
-A>:
CD
8
52.4
39.2
Table 2. Summary of Current Gating Recommendations for Assay Harmonization.
Recommendation
Cytokine gate Cytokine gate should not include negative cells and should include all of the cytokine positive cells (both high and low positive cells).
Double Positive cells Gate on CD4 and CD8 to allow clear identification of the double positive cells. Include as a separate population or with either CD4 or CD8 cells but not with both.
Dim cells Draw gates around the main populations of CD3, CD4 and CD8 that incude dim events.
Lymphocyte gate Lymphocyte gate should be large enough to include all lymphocytes. It should not include RBC or debris.
Uniform gates Uniform gates should be used for all samples within a donor.
Biexponential scaling Proper adjustment of biexponential scaling is important to clearly and completely visualize all populations.
Phase I Phase II
Your Lab Response Overall Results Your Lab Response Overall Results
CD+ Reagent Donor Media
n 25th Pctl
75th Pctl
Median
25th Pctl
75th Pctl
CD4+Unstimulated 1
Below Average 0.00 0.03 0.01 0.07
Above Average 0.05 0.02 0.02 0.04
2Below
Average 0.00 0.03 0.01 0.05 Average 0.02 0.02 0.02 0.03 3 Average 0.01 0.03 0.01 0.06 Average 0.01 0.02 0.01 0.03
CEF 1Below
Average 0.00 0.05 0.02 0.10 Average 0.05 0.04 0.03 0.05
2Below
Average 0.01 0.03 0.02 0.07 Average 0.02 0.02 0.02 0.03
3Below
Average 0.01 0.05 0.02 0.07 Average 0.03 0.04 0.03 0.04
CMVpp65 1Below
Average 0.01 0.35 0.22 0.47Below
Average 0.04 0.23 0.19 0.30
2Below
Average 0.01 0.03 0.01 0.07 Average 0.03 0.02 0.02 0.03
3Below
Average 0.01 0.07 0.04 0.10 Average 0.04 0.05 0.04 0.06
CD8+Unstimulated 1 Average 0.02 0.04 0.01 0.11
Above Average 0.05 0.03 0.02 0.04
2 Average 0.04 0.05 0.03 0.10 Average 0.03 0.03 0.03 0.05 3 Average 0.06 0.09 0.05 0.16 Average 0.08 0.06 0.05 0.09
CEF 1Below
Average 0.09 0.20 0.12 0.33Below
Average 0.05 0.17 0.13 0.23
2Below
Average 0.30 0.80 0.70 0.88Below
Average 0.31 0.73 0.69 0.76
3 Average 0.20 0.28 0.20 0.37Below
Average 0.21 0.26 0.22 0.29
CMVpp65 1Below
Average 0.06 0.24 0.15 0.39Below
Average 0.04 0.16 0.14 0.20 2 Average 0.04 0.05 0.02 0.10 Average 0.03 0.03 0.02 0.04
3 Average 0.13 0.20 0.13 0.27Below
Average 0.11 0.18 0.15 0.21
Your performance report
Phase I Review Parameter Reviewed Reviewer AssessmentAll gates uniform across all donors? Yes, always uniform.All gates uniform within each donor? Yes
Biexponential scaling correctly applied?No, populations are pushed against the axis.
Dump Negative Gate Gate includes dump positive cells.Lymphocyte Gate Gate too large, includes non-lymphocyte cells.CD3 Gate CD3 gate not too tight, includes dim cells.
CD4+ CD8+ (Double Positive) CellsDP cells are included with both CD4 and CD8 cells.
CD4 Gate CD4 Gate not too tight, dims are included.CD8 Gate CD8 Gate not too tight, dims are included.CD4+ Cytokine+ Gate: Negative Cells Gate does not include cytokine negative cells.CD4+ Cytokine+ Gate: Positive Cells Gate includes all cytokine positive cells.
CD4+ Cytokine+ Gate: Gate Proximity Gate does not include negative cells & includes all cytokine positive cells (high and low positives).
CD8+ Cytokine+ Gate: Negative Cells Gate does not include cytokine negative cells.CD8+ Cytokine+ Gate: Positive Cells Gate includes all cytokine positive cells.
CD8+ Cytokine+ Gate: Gate Proximity Gate does not include negative cells & includes all cytokine positive cells (high and low positives).
Reviewer Comments – Reviewer IDifficult to determine gating strategy from powerpoint slides.
Reviewer Comments – Reviewer II No comment provided.
Your performance report
Elispot Panel 1
Assay example: all participants testing same donor for same response
4/36 labs outliers = missed to detect > 50% of responses17/36 labs missed to detect low responder
Variability, outliers, sub-optimal performance
36 labs used own protocol, same PBMC, same antigen
Initial Elispot Harmonization Guidelines
A. Establish lab Elispot SOP for: A1. Counting method for apoptotic cells in order to determine adequate cell dilution for plating A2. Overnight resting of cells prior to platingB. Use only pretested serum with optimal signal:noise ratioC. Establish SOP for plate reading, including: C1. Human auditing during reading process C2. Adequate adjustment for technical artifactsD. Only let well trained personnel conduct assay
Janetzki et al., CII. 2008 Mar;57(3):303-15
Impact of repeated proficiency panels and ongoing harmonization (panel 1-4)
Percentage of labs that missed weak responder dropped dramatically through Panels 1, 2, and 4.
Adapted from van der Burg et al, Science Translational Medicine 2011; V3, Issue 108
Series1
0%
10%
20%
30%
40%
50%
47%
14%
7%
Serum pretested 66% (25)
No serum used 26% (10)
Serum not pretested 8% (3)
> 8 hours 66% (25)
< 8 hours 21% (8)
No resting 13% (5)
Increasing harmonization across labs (panel 5)
• All labs follow at least 2 harmonization guidelines• 68% of labs follow 4-6 harmonization guidelines
0 50 100 150 2000
20
40
60
80
100
120
140
160
180
200
Spots per 2x105 cells
CV
Overall Variability decreased by 20%(for the first time ever)
EPP4EPP5
Num
ber o
f det
ecte
d re
spon
ses
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 27 28 29 30
Lab ID
0
1
2
3
45
6
7
8
9
66% of labs were not able to detect 8 or 9 responses
Multimer Panel – Same Experience
Cancer Immunol Immunother. 2009 Oct;58(10):1701-13.
Cedrik Britten & Pedro Romero
Panel 1:
Initial Harmonization Guidelines – Multimer Staining
Britten et al. 2009 CII
Multimer Panel - MPP2 – Main Findings 1
1. Significantly higher response detection rate (e.g. 88% for all CMV responses [which were low responses])
DonorDump
Channel
Dead Cell Staining N Median
1 No No 6 0.04
No Yes 14 0.02
Yes No 6 0.04
Yes Yes 14 0.01
2 No No 6 0.06
No Yes 14 0.03
Yes No 6 0.05
Yes Yes 14 0.02
2. Dump channel& dead cellstainingdecrease noisereported
Attig et al. JTM 2011
Multimer Panel - MPP2 – Main Findings 2
3. Confirmation of harmonization guidelines
VariableNumber of Replicates
Number of Responses
% of Responses
Number of <100,000 292 146 50%CD8-
positive 100,000-150,000 164 118 72%
cells >150,000 104 94 90%Number of 2 28 10 36%
Colors 3 224 126 56%Used 4 196 142 72%
5 98 68 69%6 14 12 86%
The 14 labs that participated in both Multimer Panel 2 and Panel 3 were the top performers in Panel 3
Mul
timer
(PE)
CD8
0.301
3.97e-30.0377
99.7
D3 D2
Mul
timer
(PE)
CD8
Mul
timer
(APC
)
CD8
D4
Mul
timer
(PE)
CD8
Mul
timer
(PE)
Mul
timer
(PE)
CD8CD8
0.301
3.97e-30.0377
99.7
D3 D2
Mul
timer
(PE)
CD8
Mul
timer
(PE)
Mul
timer
(PE)
CD8CD8
Mul
timer
(APC
)
CD8
D4
Mul
timer
(APC
)
CD8
Mul
timer
(APC
)M
ultim
er (A
PC)
CD8CD8
D4
Repeated Panel participation: Road to successLabs Detected Total Number
of LabsNumberof Labs
Frequenciesof Labs
11 – 12 Responses 33 15 45.5%10 – 6 Responses 33 9 27.3%<6 Responses 33 8 24.2%
SAMPLE ASSAY ACQUISITION & ANALYSIS RESULTS
o Anticoagulanto Processingo Storageo Qualityo etc.
o Antibodieso Fluorochromeso Lyoplateso Incubation timeso Washing stepso etc.
o Machine setupo Event countso Compensationo Transformationo Gatingo etc.
o Example of Raw data
o Statisticso Response
Determinationo etc.
CELLULAR ASSAYS - SOURCES OF VARIATION
ALL THESE COMPONENTS HAVE BEEN SHOWN TO IMPACT ON ASSAY PERFORMANCE
LAB ENVIRONMENT
o Operators and Assays, SOPs, Qualification, Validation, Certification, etc.
Panel activities
SAMPLE ASSAY ACQUISITION & ANALYSIS RESULTS
CELLULAR ASSAYS - SOURCES OF VARIATION
LAB ENVIRONMENT
MANY PUBLICATIONS LACK CRITICAL INFORMATION ON THESE COMPONENTS
THIS PRECLUDES COMPARABILITY OF RESULTS GENERATED ACROSS LABS
MIATA – Initiation
42
Guidelines Version 0.0
• Published at website• Announced in
Immunity (2009, Vol. 31, p527)
Kick-off by a core team and public announcement of Version 0in October 2009
MIATA – Project Overview
43
• Public Consultation Period
• Expert Public Workshop• Three Webinars
Version 0.0• Announced in Immunity
(2009)• Published on MIATA website
Version 1.0• Published on MIATA website
(2010)
Version 2.0• Published in Immunity
(2012) • Published with additional
tools on MIATA website
• Public Consultation Period
• Expert Public workshop• Discussion with experts• Discussion with groups • Discussion with editors
Level of Acceptance
The Sample The Assay Data Acquisition Results Lab Environment
Minimal Information About:
Manuscript: Materials and Methods
Hall of Fame
Transparency
Critical protocol variables
Improved interpretation of results
MIATA stamp
Increased visibility of published work
Increased citation rate
44
Relaunch of Homepage – HALL OF FAME
HOW to implement MIATA
45
FIRST STEP SECOND STEP
visit miataproject.org
Use the checklist
Structure M&M part according to MIATA
STILL UNCERTAIN ?
THREE STEPTS TOWARDS IMPLEMENTATION
46
FIRST STEP SECOND STEP THIRD STEP
visit miataproject.org
Use the checklist
Structure M&M part according to MIATA
Check example reports
STILL UNCERTAIN ?
THREE STEPTS TOWARDS IMPLEMENTATION
47
FIRST STEP SECOND STEP THIRD STEP
visit miataproject.org
Use the checklist
Structure M&M part according to MIATA
Check example reports
FOURTH STEP
Ask Cedrik & Sylvia
WHERE to implement MIATA
48
• You can implement MIATA in every journal
• Three journals offcially recommend use if MIATA (optional choice)
o Cancer Immunotherapy Immunology
o OncoImmunology
o Cancer Immunity
• Journals may label your mansucripts with the MIATA logo on the cover page
STILL NOT CONVINCED ?
WHY to implement MIATA
49
• Accepted manuscripts that adhere to MIATA will be listed at the homepage
• Show that you control the variables critically impacting on quality of results
• This may attract more readers to your study
• Peers may acknowledge that you transparently report your experiments
• This may lead to more citations on the long-run
STILL NOT CONVINCED ?
MIATA – PRIZE FOR FIRST PUBLICATIONS !
50
CIMT is donating a prize to the first 6 manuscripts for MIATA-adherent studies
o Acceptance in a peer reviewed journal
o PRIZE goes to one member of the author list
o Study needs to adhere to MIATA
o Study needs to mention MIATA
o 500,- Euro plus free registration for upcoming CIMT meeting
TRY MIATA !!!