Date post: | 22-Dec-2015 |
Category: |
Documents |
View: | 215 times |
Download: | 0 times |
A Blind AnalysisA Blind Analysis
You are not allowed to peek!You are not allowed to peek!
Prof. Donald Koetke
Senior Research Professor of Physics
Valparaiso University
3/30/07 VU Colloquium 2
OutlineOutlineOutlineOutline
1. What is a “blind analysis”?
• What is a “bias”?
• Does “bias” = “systematic error”?
• What is the special (subtle) bias?
2. What is the history of the blind analysis”?
3. What are examples of “blind analysis” in physics, astronomy, astrophysics, and, … ?
4. Some thoughts and reflections --
3/30/07 VU Colloquium 3
OutlineOutlineOutlineOutline
1. What is a “blind analysis”?
• What is a “bias”?
• Does “bias” = “systematic error”?
• What is the special (subtle) bias?
2. What is the history of the blind analysis”?
3. What are examples of “blind analysis” in physics, astronomy, astrophysics, and, … ?
4. Some thoughts and reflections --
3/30/07 VU Colloquium 4
OutlineOutlineOutlineOutline
1. What is a “blind analysis”?
• What is a “bias”?
• Does “bias” = “systematic error”?
• What is the special (subtle) bias?
2. What is the history of the blind analysis”?
3. What are examples of “blind analysis” in physics, astronomy, astrophysics, and, … ?
4. Some thoughts and reflections --
3/30/07 VU Colloquium 5
OutlineOutlineOutlineOutline
1. What is a “blind analysis”?
• What is a “bias”?
• Does “bias” = “systematic error”?
• What is the special (subtle) bias?
2. What is the history of the blind analysis”?
3. What are examples of “blind analysis” in physics, astronomy, astrophysics, and, … ?
4. Some thoughts and reflections --
3/30/07 VU Colloquium 6
OutlineOutlineOutlineOutline
1. What is a “blind analysis”?
• What is a “bias”?
• Does “bias” = “systematic error”?
• What is the special (subtle) bias?
2. What is the history of the blind analysis”?
3. What are examples of “blind analysis” in physics, astronomy, astrophysics, and, … ?
4. Some thoughts and reflections --
3/30/07 VU Colloquium 7
OutlineOutlineOutlineOutline
1. What is a “blind analysis”?
• What is a “bias”?
• Does “bias” = “systematic error”?
• What is the special (subtle) bias?
2. What is the history of the blind analysis”?
3. What are examples of “blind analysis” in physics, astronomy, astrophysics, and, … ?
4. Some thoughts and reflections --
3/30/07 VU Colloquium 8
A Blind AnalysisA Blind AnalysisA Blind AnalysisA Blind Analysis
What is a “bias”?
Does “bias” = “systematic error”?
What is the special (subtle) bias?
“A prejudice in favor of or against” -- may bedue to computer codes, equipment performanceor setting, decisions/selections/cuts imposed, etc.
Yes - if it causes a systematic shift in a result(This is not a “mistake” or “blunder”.)
Experimenter bias in making decisions/choices to achieve a desired answer. Examples …
3/30/07 VU Colloquium 9
A Blind AnalysisA Blind AnalysisA Blind AnalysisA Blind Analysis
1. You know what the answer “should be” -i.e., you know what the “accepted” answer is -- but your answer is different -outside of errors.What do you do?
2. You make a measurement and within errorsit agrees with the “accepted” answer -- What do you do?
You do an experiment and…
Asymmetry!!
3/30/07 VU Colloquium 10
A Blind AnalysisA Blind AnalysisA Blind AnalysisA Blind Analysis
The goal of a “blind analysis” is to preventthe experimenter from (unconsciously - or consciously) making decisions in the analysis that would affect the result based on:
• The predictions of a model or theory (e.g., the Standard Model for particle physics)
• Previous measurements known to the experimenter
• The experimenter’s intuition or other predisposition
These are all examples of personal bias
3/30/07 VU Colloquium 11
A Blind AnalysisA Blind AnalysisA Blind AnalysisA Blind Analysis
1. What is a “blind analysis”?
A “blind analysis” is an analysis of measured data in which the final answer is kept hidden from the experimenters until all of the decisions about the analysis have been made:• Computer codes have been developed and tested
• Decisions about the number of events (trials) needed have been made (cuts have been selected)
• Apriori agreements are reached about what to do when the real answers from the experiment are revealed; no further analysis!
3/30/07 VU Colloquium 12
OutlineOutline
1. What is a “blind analysis”?
• What is a “bias”?
• Does “bias” = “systematic error”?
• What is the special (subtle) bias?
2. What is the history of the blind analysis”?
3. What are examples of “blind analysis” in physics, astronomy, astrophysics, and, … ?
4. Some thoughts and reflections --
3/30/07 VU Colloquium 13
HistoryHistoryHistoryHistory
Blind analysis begins in ~1930s with -
Medical research -- blind tests!
Patients don’t know whether they are getting --
a) the medicine/treatment, or,
b) a placebo
Patients are assigned to (a) or (b) ramdomly
Therefore - the patients are “Therefore - the patients are “blindblind” to their treatment” to their treatmentTherefore - the patients are “Therefore - the patients are “blindblind” to their treatment” to their treatment
They may imagine symptoms or cure, etc., but only the researcher/physician knows if these can be due to the medicine/treatment.
3/30/07 VU Colloquium 14
HistoryHistoryHistoryHistory
Blind analysis begins in ~1930s with -
Medical research -- blind tests!
• Patients talk to medical researcher
e.g., how are you feeling? is pain less or more? are the other problems? serious or not so serious? before or after you take the medicine? …etc.
• Researcher hears what patient says
• Researcher examines the patient
• Researcher records all this as “data”
Do you see a Do you see a problem here?problem here?
Do you see a Do you see a problem here?problem here?
c.f.,1937 JAMA 26 June 2178/2.
3/30/07 VU Colloquium 15
HistoryHistoryHistoryHistory
Blind analysis begins in ~1930s with -
Medical research -- blind tests!
• Therefore, the medical researcher can (and will) unconsciously and unintentionally interpret and record the information (data) with this bias (knowledge).
The patients are “The patients are “blindblind” to their treatment - ” to their treatment - butbut,,
the researcher is the researcher is NOTNOT!!
The patients are “The patients are “blindblind” to their treatment - ” to their treatment - butbut,,
the researcher is the researcher is NOTNOT!!
The solution is…
3/30/07 VU Colloquium 16
Medical research -- The double blind test!
HistoryHistoryHistoryHistory
Blind analysis begins in ~1930s with -
• The list of patients and their treatment is prepared and maintained by someone who is NOT participating in the research. The list is sealed in the “black box”.
• The list can be retrieved from the “black box” only after all the analysis is completed.
The patients are “The patients are “blindblind” to their treatment - ” to their treatment - andand,,
the researcher is “the researcher is “blindblind” to who is getting treated!” to who is getting treated!
The patients are “The patients are “blindblind” to their treatment - ” to their treatment - andand,,
the researcher is “the researcher is “blindblind” to who is getting treated!” to who is getting treated!
1948 Am. Heart Jrnl., XXXVI, 529.
1950 Am. Jrnl. Med., IX, 142/1.
3/30/07 VU Colloquium 17
Medical research -- The double blind test!
HistoryHistoryHistoryHistory
Blind analysis begins in ~1930s with -
Consider a simple analysis -- “Was the medicine effective?”
• The data on all patients will indicate whether the medication was effective for each patient.
• When this data analysis is complete (and free from mistakes) and any disagreement among the researchers have been settled and any concerns about bias have been removed, i.e., then (and only then) -
• The “black box” can be opened and the list can be retrieved.
• The data on all patients is now grouped into two groups: group (a) and group (b)
3/30/07 VU Colloquium 18
Medical research -- The double blind test!
HistoryHistoryHistoryHistory
Blind analysis begins in ~1930s with -
The simple analysis -- “Was the medicine effective?”
• For what fraction of group (a) [fa] and group (b) [fb] was the medicine effective?• Do a statistical analysis to determine whether the difference in these two fractions is statistically significant - or whether it is consistent with a random occurrence. The result is now known!
• The researchers agree to accept the results obtained and no further analysis of the data is permitted - unless there has been a blunder (mistake) - in which case fix the blunder and report both results and the nature of the “fix”.
3/30/07 VU Colloquium 19
HistoryHistoryHistoryHistory
Blind analysis has become the standard methodology in clinical trials.
Blind analysis has been used in the physical sciences only in recent years.
Physicists (and astronomers) are, of course ---- Careful-- Quantitative-- Attentive to bias (systemtic errors)
Why should they need a blind analysis?
3/30/07 VU Colloquium 20
HistoryHistoryHistoryHistory
Ernest Rutherford (1934) --
“It seems to me that in some way it is regrettable that we had a theory of the positive electron before the beginning of the experiments. Blackett* did everything possible not to be influenced by the theory, but the way of anticipating results must inevitably be influenced to some extent by the theory. I would have liked it better if the theory had arrived after the experimental facts had been established.” Ernest Rutherford, Proc. Solvay Conference,
(Gauthier-Villars, Paris 1934), p 177.* Nobel prize in physics for discovering the positron (1948)
3/30/07 VU Colloquium 21
HistoryHistoryHistoryHistory
Gregor Mendel (1865) --
The classic case is Gregor Mendel’s work on inheritance.
84 degrees of freedom, giving a probability that hewould have measured this well to be only 7x10-5 !!
In every case the data agreed with the theoretical ratios within less than the standard errors.Taking the whole together, 2 was 41.6 on
3/30/07 VU Colloquium 22
HistoryHistoryHistoryHistory
Modern examples --
From nuclear and subnuclear physics ---
Reason: It’s the field in which the blind analysis techniques have been widely used - and -
It is the field with which I am most familiar.
3/30/07 VU Colloquium 23
HistoryHistoryHistoryHistory
3/30/07 VU Colloquium 24
HistoryHistoryHistoryHistory
3/30/07 VU Colloquium 25
B-meson lifetime ratio
The mean ratio has a 2 of 4.5 for 13 degrees of freedom; P=0.985
8 Major experiments
Over 2000 physicists
Bottom line:
The agreement appears to be
too good!!
HistoryHistoryHistoryHistory
3/30/07 VU Colloquium 26
OutlineOutlineOutlineOutline
1. What is a “blind analysis”?
• What is a “bias”?
• Does “bias” = “systematic error”?
• What is the special (subtle) bias?
2. What is the history of the blind analysis”?
3. What are examples of “blind analysis” in physics, astronomy, astrophysics, and, … ?
4. Some thoughts and reflections --
3/30/07 VU Colloquium 27
An example of a “search” for an uncommon occurrence
A reaction that violates the Standard Modelconservation of lepton number
If you do not find the reaction, you have not shown the Standard Model to be incorrect
If you do find the reaction, you had better be very sure that you have got it right!!!
3/30/07 VU Colloquium 28
MEGA
Search for e
Question: When and How are the values of determined?
Signal: Ee = E = 52.8 MeV
Boxes represent 2 boundaries
ExamplesExamplesExamplesExamples
3/30/07 VU Colloquium 29
MEGA
Search for e
Blind Analysis: The boundaries aredetermined before the analysis of
(most of) the data!
Signal: Ee = E = 52.8 MeV
Boxes represent 2 boundaries
ExamplesExamplesExamplesExamples
3/30/07 VU Colloquium 30
3/30/07 VU Colloquium 31
TWIST
High precision ee
ExamplesExamplesExamplesExamples
A search for physics beyond (not included in) the Standard Model.
p = 29.8 MeV/c
Measure: pe()
3/30/07 VU Colloquium 32
Nu
mb
er
X = E/Emax
N(x) = a + bxa,b from theory
ExamplesExamplesExamplesExamples
Simulation = “Monte Carlo”
Simulation ofthe experiment
using a,b
Nexp(x) = a’ + b’x
Analyzed expimental
data
Find a’ & b’ bycomparison of
real data with
simulateddata
Find a’ & b’ bycomparison of
real data with
simulateddata
The detector system distorts the distribution
3/30/07 VU Colloquium 33
Muon decay spectrum
Current SM = -0.007 ± 0.013 0 = 0.7518 ± 0.0026 3/4 = 0.7486 ± 0.0026 ± 0.0028 3/4P = 1.0027 ± 0.0079 ± 0.0030 1
)1(3)34(3
2)33(
)(cos0
2
2
xx
xxx
dxdx
d−+−+−∝
Γ
⎥⎦⎤
⎢⎣⎡ −+−+ )34(
3
2)1(cos xxP δθξμ
ExamplesExamplesExamplesExamples
3/30/07 VU Colloquium 34
EnergyCos()
ExamplesExamplesExamplesExamples
Muon decay spectrum from Standard Model
3/30/07 VU Colloquium 35
EnergyCos()
TWIST measured spectrum
ExamplesExamplesExamplesExamples
EnergyCos()
Standard Model spectrum
Monte Carlo computer code
,,,
3/30/07 VU Colloquium 36
EnergyCos()
TWIST simulated spectrum
,,,
EnergyCos()
TWIST measured spectrum
ExamplesExamplesExamplesExamples
Problem!
This spectrum is the S.M. spectrum -- nothing is
hidden; the experimenter is not “blinded”.
Compare spectra ’,’,’,’
’,’,’,’
3/30/07 VU Colloquium 37
EnergyCos()
TWIST measured spectrum
ExamplesExamplesExamplesExamples
EnergyCos()
Standard Model spectrum
Monte Carlo computer code
o,o,o,o
Unknown spectrum
o,o,o,o are generated randomly, are encrypted, stored secretly, and used to generate the simulated data. Nobody knows what the offsets from ,,, are.
o,o,o,o are generated randomly, are encrypted, stored secretly, and used to generate the simulated data. Nobody knows what the offsets from ,,, are.
3/30/07 VU Colloquium 38
EnergyCos()
TWIST simulated spectrum
?
EnergyCos()
TWIST measured spectrum
ExamplesExamplesExamplesExamples
Compare spectra to get , , ,
’,’,’,’
= ’ - o
= ’ - o = ’ - o = ’ - o
3/30/07 VU Colloquium 39
ExamplesExamplesExamplesExamples
Before you can open the black box:
• Get all computer codes working and tested
• Identify all sources of systematic error and evaluate the size of each one
• Take all of the data you will need including data to help estimate the systematic errors
• Be sure that the Monte Carlo programs accurately simulate your experiment or you will have false values
• Analyze all of the data you intend to use to get the result
o,o,o,o
3/30/07 VU Colloquium 40
ExamplesExamplesExamplesExamples
o,o,o,o = ’ - o
= ’ - o = ’ - o = ’ - o
MeasureCalculate the results
Compare with Standard Model predictions
Write the paper
3/30/07 VU Colloquium 41
ExamplesExamplesExamplesExamples
Let’s look at one of your experiments Let’s look at one of your experiments
from PHYS-245from PHYS-245
3/30/07 VU Colloquium 42
ExamplesExamplesExamplesExamples
Present accepted value: c = 299,792,458 m/sc 3 x 108 m/s
A measurement of the speed of light
12
€
T
osc
€
t =θ
2πT ⇒ c =
D
t
D
3/30/07 VU Colloquium 43
ExamplesExamplesExamplesExamples
Present accepted value: c = 299,792,458 m/sc 3 x 108 m/s
A measurement of the speed of light
12
€
T
osc
€
t =θ
2πT ⇒ c =
D
t
D
€
To = T +δT
€
To
€
co
Blinded
3/30/07 VU Colloquium 44
OutlineOutlineOutlineOutline
1. What is a “blind analysis”?
• What is a “bias”?
• Does “bias” = “systematic error”?
• What is the special (subtle) bias?
2. What is the history of the blind analysis”?
3. What are examples of “blind analysis” in physics, astronomy, astrophysics, and, … ?
4. Some thoughts and reflections --
3/30/07 VU Colloquium 45
ReflectionsReflectionsReflectionsReflections
1. Guide to a blind analysis: If my answer were to come out to be six standard deviations from the expected result, what would I do? Make the list, and then -do all of that before you look at the answer!
2. A blind analysis is intended to guard against experimenter bias. It will not guard against fraud; that requires integrity and honesty.
3. A blind analysis is likely not the fastest way to an answer.
3/30/07 VU Colloquium 46
ReflectionsReflectionsReflectionsReflections
4. It is not always possible to achieve perfect “blindness” -- e.g., drug testing.
5. Don’t need to plan everything in the analysisbefore beginning; just keep the answer hidden.
6. A blind analysis may not work for every experiment -but it is worth investigating before you begin.
7. In a blind analysis you want to hide the answer from anyone else who might want to offer advice that may be based on the answer.
3/30/07 VU Colloquium 47
Thankyou
ReflectionsReflectionsReflectionsReflections
8. A blind analysis removes the “comfort factor” - knowing what answer you are getting so you can make changes, do more analysis, repeat measurements, etc., if the answer is not what you expect.
But, that may not be the best science!But, that may not be the best science!But, that may not be the best science!But, that may not be the best science!
3/30/07 VU Colloquium 48