1/17/13 4:13 PM 1
Experimental Design for Biologists
Lecture Topics
A. A little history & philosophy B. Frameworks C. System Design; System Analysis
D. Experimental Controls
E. Model Building & Verification
2
Experiment 2
Experimental Program: ... the process of doing science
Experiment 2
Experiment 1
Project A
Experiment 1
Project B
Experiment 1
Experiment 2
Experimental Program: Includes Distinct Projects, performed under particular “Frameworks”
Project B
Experiment 1
Experiment 2
FRAMEWORK
Experiment 1
Experiment 2
Project A
Experiment 1
Experiment 2
FRAMEWORK
Experimental Program: Includes Distinct Projects, performed under particular “Frameworks”
Project B
Experiment 1
Experiment 2
FRAMEWORK
Experiment 1
Experiment 2
Project A
Experiment 1
Experiment 2
FRAMEWORK Constructed by Philosophy
1/17/13 4:13 PM 6
What is a Framework?
Refers to multiple aspects of the project...
1) Why is the experiment being considered? 2) What type of experiments need to be done? 3) How will the experiment be designed? 4) How will the data be analyzed? 5) What can be obtained from an experiment? 6) How does one use the experimental results?
1/17/13 4:13 PM 7
Beginning of “Modern Science”
Galileo: Emphasis on the Experiment, even though in his case it was mostly ‘thought experiments”, as a way of explaining an idea
However, his process was more in line with Aristotle/Archimedes View of the world, leading to experiment/observation, conclusion
Bacon: Emphasis on the Experiment, as a way of building a model - then
generalization via “induction”.
Suspicion of Intuition, or current views as to how things work.
Legalistic/process driven approach, as opposed to requirement for absolute, mathematical certainty (thus perhaps more appropriate for biology).
1/17/13 4:13 PM Experimental Design
8
What is Inductive Reasoning?
Two definitions of induction: prediction of the future; generalization to a broader case Emphasized by Francis Bacon, in “Novum Organum” (1621) and then again by Isaac Newton in the Principia, and in his Optiks Inductive Reasoning: Application of finding X to “broader” or “general” case - as in the future case.
1/17/13 4:13 PM 9
Francis Bacon on Inductive Reasoning
XIV. The syllogism consists of propositions, propositions consist of words, words are symbols of notions. Therefore if the notions themselves (which is the root of the matter) are confused and over-hastily abstracted from the facts, there can be no firmness in the superstructure. Our only hope therefore lies in a true induction.
XIX. There are and can be only two ways of searching into and discovering truth. The one flies from the senses and particulars to the most general axioms, and from these principles, the truth of which it takes for settled and immovable, proceeds to judgment and to the discovery of middle axioms. And this way is now in fashion. The other derives axioms from the senses and particulars, rising by a gradual and unbroken ascent, so that it arrives at the most general axioms last of all. This is the true way, but as yet untried.
NOTE: Critique of a Hypothesis-based methodology, but in the context of VERIFICATION of an unproven premise, vs evidence-based model-building.
1/17/13 4:13 PM 10
Newton on the Hypothesis
Isaac Newton started with hypotheses, but came to reject them, accepting Bacon’s approach… Newton: Second edition of Principia:
Hypotheses non fingo: “Hypotheses I make not”
Optiks: “Hypotheses have no place in experimental science.” The idea was that one should not establish something as a hypothesis in advance of an experiment, or an explanation; and once you had such evidence, it was no longer “hypothetical”.
1/17/13 4:13 PM 11
David Hume - early 1700s, after Bacon & Newton
“Treatise of Human Nature”: expressed extreme skepticism that the past could be said to rationally predict the future Held that one could only know what was true in observed phenomena, meaning via experience... ...but experience is not applicable to the future, according to Hume. Thus he rejected Inductive Reasoning.
1/17/13 4:13 PM 12
Hume’s Rejection of Inductive Reasoning 1739, David Hume wrote a Treatise of Human Nature, and introduced “radical skepticism” - a rejection of probability derived from past experience as predicting the future. “Your appeal to past experience decides nothing in the present case; and at the utmost can only prove, that that very object, which produced any other, was at that very instant endowed with such a power; but can never prove, that the same power must continue in the same object or collection of sensible qualities; much less, that a like power is always conjoined with like sensible qualities. Should it be said, that we have experience, that the same power continues united with the same object,and that like objects are endowed with like powers, I would renew my question, why from this experience we form any conclusion beyond those past instances, of which we have had experience.” Hume thus rejected “Inductive Reasoning”, that from the experience A causes B, we can infer that in the future A’ will cause B’, saying that the “future A’ ” may not resemble the present A.
1/17/13 4:13 PM 13
Hume’s Rejection of Inductive Reasoning gave rise to Popper’s Hypothesis, which was different (falsification)
Popper proposes “Critical Rationalism” - and suggests the following: 1. Verification cannot be assumed from past experience. 2. Falsification only can be demonstrably shown.
Question to think about: If one is truly limited to falsification, how can one rationally build on a past experiment, so as to design a follow-up? (Popper’s answer & Hume’s answer)
1/17/13 4:13 PM 14
The Dominant Framework of Scientific Experimentation involves the “testing of falsifiable hypotheses”
1. Falsifiable Hypothesis - developed by Karl Popper, under a theory called Critical Rationalism.
1/17/13 4:13 PM 15
The Dominant Framework of Scientific Experimentation involves the “testing of falsifiable hypotheses”
1. Falsifiable Hypothesis - developed by Karl Popper, under a theory called Critical Rationalism. 2. Hypothesis a) statement about the observable universe that is formulated in such a manner that it can be tested.
b) it is a tentative explanation of events or of how something works.
1/17/13 4:13 PM 16
1. Falsifiable Hypothesis - developed by Karl Popper, under a theory called Critical Rationalism. 2. Hypothesis a) statement about the observable universe that is formulated in such a manner that it can be tested.
b) it is a tentative explanation of events or of how something works.
3. Key component - the statement must be falsifiable.
Oft unmentioned requirement of the hypothesis under Critical Rationalism - rejection of verification
The Dominant Framework of Scientific Experimentation involves the “testing of falsifiable hypotheses”
1/17/13 4:13 PM 17
Advantages of the Hypothesis, when used in the sense of falsification
1. Forces restriction of experiments to ideas that can be falsified (establishes what is testable) 2. Instills a sense of skepticism, which may be helpful in critically analyzing data 3. Teaches the concept of a “framework” - that the data one produces might be limited in their applicability
1/17/13 4:13 PM 18
Published Criticisms of Critical Rationalism
1. Critical Rationalism does not avoid inductive reasoning. 2. Extreme skepticism can be paralyzing, if it does not allow one to say it is rational to look at the past as a predictor of the future. (identity ex) 3. The past is a rational predictor of the future, since there is nothing about time which would give a mechanism as to why “laws of nature” should start working differently at a particular point. 4. “Falsificationalism” gets you nowhere.
1/17/13 4:13 PM 19
Statistical Hypothesis- testing is distinct from the philosophical use of the term
1. Role of statistics: to determine if there is a difference between groups of data 2. Type I statistical error: when a difference is claimed when there is none (false positive) 3. Type II statistical error: when a difference exists but is not recognized. (false negative) Question to think about: Role of repetition in determining statistical significance, and how repetition functions in different philosphical contexts.
1/17/13 4:13 PM 20
Statistical Hypothesis-Testing
none of the criticisms of Critical Rationalism should be confused with the term “hypothesis” as it is used in the context of Statistics. HOWEVER… it is controversial as to where statistical verification can be said to be predictive (traditional vs Beyesian statistics)
1/17/13 4:13 PM 21
Additional Criticisms of the Hypothesis as an Experimental Framework (filtration criticism)
consider this example...
1/17/13 4:13 PM 22
Hypothesis: The Sky is Red Experimental Design: First Determine the wavelength of light which causes a “red” readout, using a positive control known to be red. Set a wavelength-meter to signal when the color red is detected As a negative control set a second wavelength meter to detect any color other than red. Collect data over 24 hours. See if you got a “positive” readout. Repeat the experiment ten times times. Nine out of ten times, the color red has been positively detected. Conclusion: The sky is red (or the hypothesis is not falsified)
1/17/13 4:13 PM 23
Hypothesis: The Sky is Red Experimental Design: First Determine the wavelength of light which causes a “red” readout, using a positive control known to be red. Straw Man? Does the hypothesis force a reading of any other color except “not red?” Compare to a hypothesis where you don’t have as much inductive information: Activation of NF-kB induces inflammation Ingestion of caffeine induces the likelihood of cancer
1/17/13 4:13 PM 24
Might one conclude from this experiment that the sky is red? Issues: 1) Need to establish the “usual case” 2) Need to establish definition of terms
does the hypothesis mean “red appears in the sky?” or does it mean “the sky is usually red?”
3) Scientific disagreements are often about “semantics”
Semantics are important
1/17/13 4:13 PM 25
The Hypothesis “the sky is red” might have biased the conclusion in the following ways: 1) It demanded only one type of measurement
1/17/13 4:13 PM 26
The Hypothesis “the sky is red” might have biased the conclusion in the following ways: 1) It demanded only one type of measurement 2) It caused all other color measurements to be grouped as “negative data”, since they did not prove the hypothesis “true” - establishing a binary “positive”/”negative” distinction
1/17/13 4:13 PM 27
The Hypothesis “the sky is red” might have biased the conclusion in the following ways: 1) It demanded only one type of measurement 2) It caused all other color measurements to be grouped as “negative data”, since they did not prove the hypothesis “true” - establishing a binary “positive”/”negative” distinction 3) It allowed for validation(or lack of falsification) even though the result did not represent the “normal” situation (it did not control for rare or “usual” events) (question: does this matter)
1/17/13 4:13 PM 28
The Hypothesis “the sky is red” might have “biased” the conclusion in the following ways: 1) It demanded only one type of measurement 2) It caused all other color measurements to be grouped as “negative data”, since they did not prove the hypothesis “true” - establishing a binary “positive”/”negative” distinction 3) It allowed for validation(or lack of falsification) even though the result did not represent the “normal” situation (it did not control for rare or “usual” events) (question: does this matter) 4) Sampling times were not altered (it did not control for hidden variables, like color changes within the 24 hour time period)
1/17/13 4:13 PM 29
The Hypothesis “the sky is red” might have biased the conclusion in the following ways: 1) It demanded only one type of measurement 2) It caused all other color measurements to be grouped as “negative data”, since they did not prove the hypothesis “true” - establishing a binary “positive”/”negative” distinction 3) It allowed for validation(or lack of falsification) even though the result did not represent the “normal” situation (it did not control for rare or “usual” events) (question: does this matter) 4) Sampling times were not altered (it did not control for hidden variables, like color changes within the 24 hour time period) 5) Question: Can the conclusion as stated be criticized?
1/17/13 4:13 PM 30
The Hypothesis “the sky is red” might have biased the conclusion in the following ways: 1) It demanded only one type of measurement 2) It caused all other color measurements to be grouped as “negative data”, since they did not prove the hypothesis “true” - establishing a binary “positive”/”negative” distinction 3) It allowed for validation(or lack of falsification) even though the result did not represent the “normal” situation (it did not control for rare or “usual” events) (question: does this matter) 4) Sampling times were not altered (it did not control for hidden variables, like color changes within the 24 hour time period) 5) Question: Can the conclusion as stated be criticized? 6) Exercise: compare to a more familiar biological hypothesis, such as “Raf and akt synergize”. Does this hypothesis escape the problems of the “sky is red” hypothesis.
1/17/13 4:13 PM 31
Other issues with framing experiments with hypotheses: 1) The hypothesis mirrors the conclusion: Hypothesis: The Sky is Red Conclusion: The Sky is Red This formulation can make the experimenter confuse the unproven with the proven
1/17/13 4:13 PM 32
Other issues with framing experiments with hypotheses: 1) The hypothesis mirrors the conclusion: Hypothesis: The Sky is Red Conclusion: The Sky is Red This formulation can make the experimenter confuse the unproven with the proven 2) The hypothesis by definition is a statement without inductive basis - odds of “correctness” therefore low
1/17/13 4:13 PM 33
Other issues with framing experiments with hypotheses: 1) The hypothesis mirrors the conclusion: Hypothesis: The Sky is Red Conclusion: The Sky is Red This formulation can make the experimenter confuse the unproven with the proven 2) The hypothesis by definition is a statement without inductive basis - odds of “correctness” therefore low 3) Falsification is not rewarded
1/17/13 4:13 PM 34
Other issues with framing experiments with hypotheses: 1) The hypothesis mirrors the conclusion: Hypothesis: The Sky is Red Conclusion: The Sky is Red This formulation can make the experimenter confuse the unproven with the proven 2) The hypothesis by definition is a statement without inductive basis - odds of “correctness” therefore low 3) Falsification is not rewarded 4) Experiments are aimed at confirmation, because of the requirement to measure the “positive” result - therefore there is a tension between the experiment and the philosophical paradigm framing the experiment
1/17/13 4:13 PM 35
Other issues with framing experiments with hypotheses: 1) The hypothesis mirrors the conclusion: Hypothesis: The Sky is Red Conclusion: The Sky is Red Hypothesis: Raf and Akt synergize Conclusion: Raf and Akt synergize This formulation can make the experimenter confuse the unproven with the proven 2) The hypothesis by definition is a statement without inductive basis - odds of “correctness” therefore low 3) Falsification is not rewarded 4) Experiments are aimed at confirmation, because of the requirement to measure the “positive” result - therefore there is a tension between the experiment and the philosophical paradigm framing the experiment 5) The negative set may be inappropriately used as a filter that = lack of positive data, as opposed to “not positive”
1/17/13 4:13 PM 36
Settings where the Hypothesis is not practical: “Big Science”
1. Genome sequencing (what would be a hypothesis seeking to be falsified
for this experiment?) (difference between goals, motivations and hypotheses)
1/17/13 4:13 PM 37
Settings where the Hypothesis is not practical: “Big Science”
1. Genome sequencing (what would be a hypothesis seeking to be falsified
for this experiment?) (difference between goals, motivations and hypotheses)
2. Snp search vs snp test
distinction between “hypothesis producing” vs “hypothesis testing” settings
Point: the “hypothesis producing” experiment is still an experiment, and is still governed by all the necessary requirements of an experiment...
1/17/13 4:13 PM 38
Settings where the Hypothesis is not practical: “Big Science”
1. Genome sequencing (what would be a hypothesis seeking to be falsified
for this experiment?) (difference between goals, motivations and hypotheses)
2. Snp search vs snp test
distinction between “hypothesis producing” vs “hypothesis testing” settings
Point: the “hypothesis producing” experiment is still an experiment, and is still governed by all the necessary requirements of an experiment...
3. Microarray experiment vs follow-up experiments
First experiment: compare expression differences between conditions x & y (falsification is not the goal for the hypothesis that there are differences between x & y) Follow-up experiments: ask or hypothesize whether a particular difference plays a mechanistic role in the condition y
1/17/13 4:13 PM 39
“Big Science” can be framed by questions
1. Genome sequencing - what is the sequence of X?
2. Snp search - what are the sequence differences between individuals of phenotype x and y?
3. Microarray experiment - what are the expression differences
between tissue treated with x vs not treated? EXERCISE: compare these questions to hypotheses as
frameworks for this type of science - which “works” better?
1/17/13 4:13 PM 40
1. A process of iterative querying, from a broad question to a series of increasingly narrow questions, or a series of yes/no questions. Example: What color is the sky?
Is the sky red? Is the sky blue? etc
Example: How do you get to Carnegie Hall? Does route X take you to Carnegie Hall? Does route Y take you to Carnegie Hall?
Example: What is the fastest way to Carnegie Hall? Is route X faster than an other route Y, Z, etc?
Example: What are the effects of Raf? What are the effects of Raf on proliferation? Does activation of raf perturb proliferation? How does raf perturb proliferation? Does raf perturb proliferation via Erk1?
The Problem/Question as a Framework for Experimental Projects
41
It can accept the idea of inductive reasoning
Features of The Problem/Question Framework
42
It can accept the idea of inductive reasoning It does not make a distinction between “positive” or “negative” data (but models
are either verified or they are not)
Features of The Problem/Question Framework
43
It can accept the idea of inductive reasoning It does not make a distinction between “positive” or “negative” data (but models
are either verified or they are not)
It is framed with an open-ended question, which should allow an answer
Features of The Problem/Question Framework
44
It can accept the idea of inductive reasoning It does not make a distinction between “positive” or “negative” data (but models
are either verified or they are not)
It is framed with an open-ended question, which should allow an answer
The question is a different structure than the answer
Features of The Problem/Question Framework
45
It can accept the idea of inductive reasoning It does not make a distinction between “positive” or “negative” data (but models
are either verified or they are not)
It is framed with an open-ended question, which should allow an answer
The question is a different structure than the answer
Answering the question is rewarded
Features of The Problem/Question Framework
46
It can accept the idea of inductive reasoning It does not make a distinction between “positive” or “negative” data (but models
are either verified or they are not)
It is framed with an open-ended question, which should allow an answer
The question is a different structure than the answer
Answering the question is rewarded
It is somewhat more flexible: you can ask questions like “How...?” “What...?” Reframing these as hypotheses may obscure what the scientist wants to achieve from the experiment. It is appropriate for Big Science
Features of The Problem/Question Framework
1/17/13 4:13 PM 47
1. The question is a different structure than the answer
2. Can be phrased such that no answer is deemed “negative”.
3. Can accept the idea of inductive reasoning
4. Can accept the idea of verification, via model-building and tests of a model (“Is the model true?”)
5. Statistics can be used to determine model verification, based on the answer to the question
Features of The Problem/Question Framework
1/17/13 4:13 PM Experimental Design Harvard Medical School
48
Project Schematic in The Problem/Question Framework
Model
Query Model Verification?
Question Answer step 1
Question Answer step 2; repeat
step 3
Perturbed Model
1/17/13 4:13 PM Experimental Design Harvard Medical School
49
Project Schematic in Hypothesis-Testing Framework
Conclusion
Test Conclusion Lack of Falsification (is model building valid without inductive reasoning?)
Hypothesis Lack of Falsification step 1
Hypothesis Lack of Falsification
step 2; repeat
step 3
What type of conclusion is appropriate under Critical Rationalism? Only lack of falsification.
Model
Experimental Program: Includes Distinct Projects, performed under particular “Frameworks”
Project B
Experiment 1
Experiment 2
FRAMEWORK
Experiment 1
Experiment 2
Project A
Experiment 1
Experiment 2
FRAMEWORK Constructed by Philosophy
Experimental Program: Includes Distinct Projects, performed under particular “Frameworks”
Project B
Experiment 2
FRAMEWORK
Experiment 1
Experiment 2
Project A
Experiment 2
FRAMEWORK Constructed by Philosophy
Sy
Experiment 1
System
Experiment 1
System
1/17/13 4:13 PM Experimental Design Harvard Medical
School
52
System Issues Capability - Can you measure/do the thing you want to
measure/do? – answered by these other properties:
Sensitivity - How much of the thing can you measure?
To what degree of difference can you measure it? (bp ex)
Specificity/Selectivity - Can you measure/do the thing you
want such that you can discriminate between X & Y, where X is the thing you want to measure & Y are other things which might interfere with you measuring X; X & Y are the signal to noise represented by the positive & negative controls, respectively
Fidelity/Stability - Can you measure the thing consistently?
Do you get the same answer under the same conditions.
Two an'bodies for detec'ng MuSK from Santa Cruz, inc.
An'body A An'body B
Experimental Design Harvard Medical School
Two an'bodies for detec'ng MuSK from Santa Cruz, inc.
An'body A An'body B
Experimental Design Harvard Medical School
1/17/13 4:13 PM Experimental Design Harvard Medical
School
55
System Issues for an antibody Capability - Can you measure/do the thing you want to
measure/do? – answered by these other properties: (also issues like IP vs IB)
Sensitivity - How much of the thing can you measure?
To what degree of difference can you measure it? Dose down lysate or peptide
Specificity/Selectivity - Can you measure/do the thing you
want such that you can discriminate between X & Y, where X is the thing you want to measure & Y are other things which might interfere with you measuring X; X & Y are the signal to noise represented by the positive & negative controls, respectively; Peptide alone vs lysate; KO)
Fidelity/Stability - Can you measure the thing consistently?
Do you get the same answer under the same conditions. (Stability of ab; stab of lysate)
1/17/13 4:13 PM Experimental Design Harvard Medical
School
56
Examples of Systems that require validation Animal models
Genetic model Treatment model Pharmacologic model Translatability? Genetic variation? Discrete markers?
Cellular models
Genetic perturbation Treatment model Pharmacologic model Translatability to the normal cell? Inductive ability?
Biochemical/Test-tube experiments
(PCR, Westerns, Southerns etc) Conditions used Reagents
1/17/13 4:13 PM Experimental Design Harvard Medical
School
57
The use of the Positive Control establishes that the system is working.
The positive control in an experiment is a system control.
System validation is the process of understanding the
parameters of the positive control in the context of your model.
Additional issues in system validation depend on why you’re
using the system; for example, the issue of the generalizability of your results from your system to another
needs to be validated..
1/17/13 4:13 PM Experimental Design Harvard Medical
School
58
System Issues (capability, sensitivity, specificity) Need to be Validated with ALL CONTRLS Used in the Experiment
Negative Control - e.g. No antibody; irrelevant antibody Positive Control – e.g. Different, previously validated
antibody; recombinant protein to demonstrate antibody works
Sensitivity Controls – discussed further today Specificity Controls – “ Selectivity Controls - “ Thus there is a circularity to System Validation – the Positive control in an experiment is a way of asking “did the system work” each time the experiment is done.
1/17/13 4:13 PM Experimental Design Harvard Medical
School
59
Experimental Design for Biologists - Establishing the System
The Framework Question or Hypothesis Governs the Choice of System 1) The framework operates most effectively if it can govern the entire project - not just a single experiment
Once you establish what you want to answer, you can figure out what tools you need to answer that
question.
Note - some people pick the system first: Then you are limited to particular questions/hypotheses
appropriate to that system.
1/17/13 4:13 PM Experimental Design Harvard Medical
School
60
Experimental Design for Biologists - Establishing the system by accessing the “inductive space”
The choice of the system is often governed by the “inductive space” established by the framework question or hypothesis 1) The framework operates most effectively if it can govern the entire project - not just a single experiment
2) The framework helps guide the scientist as to the knowledge which might impinge on the experiment - this is referred here as “accessing the inductive space”
The “Inductive Space” is the knowledge gained from past experience, which the scientist might profit from applying to the current issue.
Challenge of knowing what to access: 1. What is relevant about the thing you are studying?
For a protein it may be a domain, or an activity, or the cellular localization, or some other thing that it perturbs, which is your readout
2. This is another reason why the framework issues are so important
- what are you actually interested in finding out - how protein X perturbs Y, or what are the things which cause Y to do Z? (you need to make sure you are studying the right thing)
1/17/13 4:13 PM Experimental Design Harvard Medical
School
62
Establishing the system
MISTAKE: MOST PEOPLE JUMP TO THIS:
Experimental question: Does caffeine cause an increase in blood pressure? Experimental hypothesis: Caffeine causes an increase in blood pressure. (Exercise: What is the framework question?)
1/17/13 4:13 PM Experimental Design Harvard Medical
School
63
Establishing the system
Correct way: Understand your framework question, so you know full range of what you’ll need to measure. Then prove you can measure this thing, and in what context (what
animal, under what conditions): that is your “System”
Framework: What is the effect of caffeine on blood pressure? (have to measure both an increase or a decrease, or a lack of change, vs only an increase) (Broader framework: “what is the effect of caffeine?” would alert thescientist to the possibility of other effects that might perturb the experimental issue, even if not measured in a particular experiment - “independent variables”, side effects) Experimental question: Does caffeine cause an increase in blood pressure? Experimental hypothesis: Caffeine causes an increase in blood pressure.
1/17/13 4:13 PM Experimental Design Harvard Medical
School
64
Framework: What is the effect of caffeine on blood pressure? (have to measure both an increase or a decrease, or a lack of change, vs only an increase) Experimental question: Does caffeine cause an increase in blood pressure? Experimental hypothesis: Caffeine causes an increase in blood pressure.
Systems: Measuring changes in blood pressure Human, Dog, Rodent? How are you going to perturb blood pressure? (pharmacologic agent?
how are you going to validate that drug?) How are you going to measure/establish what the blood pressure is?
Establishing the system
1/17/13 4:13 PM Experimental Design Harvard Medical
School
65
Framework: What is the effect of caffeine on blood pressure? (have to measure both an increase or a decrease, or a lack of change, vs only an increase) Experimental question: Does caffeine cause an increase in blood pressure? Experimental hypothesis: Caffeine causes an increase in blood pressure.
Systems: Measuring changes in blood pressure
A) Equipment
Establishing the system
1/17/13 4:13 PM Experimental Design Harvard Medical
School
66
Framework: What is the effect of caffeine on blood pressure? (have to measure both an increase or a decrease, or a lack of change, vs only an increase) Experimental question: Does caffeine cause an increase in blood pressure? Experimental hypothesis: Caffeine causes an increase in blood pressure.
System: Measuring changes in blood pressure
A) Equipment B) Equipment verification
Establishing the system
1/17/13 4:13 PM Experimental Design Harvard Medical
School
67
Framework: What is the effect of caffeine on blood pressure? (have to measure both an increase or a decrease, or a lack of change, vs only an increase) Experimental question: Does caffeine cause an increase in blood pressure? Experimental hypothesis: Caffeine causes an increase in blood pressure.
System: Measuring changes in blood pressure
A) Equipment B) Equipment verification C) Perturbation
Establishing the framework establishes the system
1/17/13 4:13 PM Experimental Design Harvard Medical
School
68
Framework: What is the effect of caffeine on blood pressure? (have to measure both an increase or a decrease, or a lack of change, vs only an increase) Experimental question: Does caffeine cause an increase in blood pressure? Experimental hypothesis: Caffeine causes an increase in blood pressure.
System: Measuring changes in blood pressure
A) Equipment B) Equipment verification C) Perturbation D) Perturbation verification
Establishing the system
1/17/13 4:13 PM Experimental Design Harvard Medical
School
69
Framework: What is the effect of caffeine on blood pressure? (have to measure both an increase or a decrease, or a lack of change, vs only an increase) Experimental question: Does caffeine cause an increase in blood pressure? Experimental hypothesis: Caffeine causes an increase in blood pressure.
System: Measuring changes in blood pressure
A) Equipment B) Equipment verification C) Perturbation D) Perturbation verification
The way one addresses these issues is with “system controls”
Establishing the system
1/17/13 4:13 PM Experimental Design Harvard Medical
School
70
Question: Does caffeine cause an increase in blood pressure? Equipment Verification Controls
A) Validation vs a standard example: blood pressure monitor: compare to “validated” standard 2nd example: pH meter: use solution of “known” pH 3rd example: antibody: test on peptide used to raise ab
B) Validation by different pieces of equipment
Do two different blood pressure monitors agree? Do two antibodies raised against the same peptide agree?
C) Reproducibility of measurement
Does the equipment give the same answer when the same condition is surveyed multiple times? What is the degree of difference?
D) Different type of equipment or different method of addressing the issue
E) Subject population can experience changes in blood pressure (what if they’re all on medication?)
Establishing the system - system controls
1/17/13 4:13 PM Experimental Design Harvard Medical
School
71
Does caffeine cause an increase in blood pressure? Perturbation Verification Controls
A) Positive Control Reagent known to cause an increase in blood pressure
B) Negative control
Reagent known to have no effect on blood pressure
C) “All but X” negative control
Decaffeinated coffee vs coffee
C) Equipment control - sensitivity of changes
What is the degree of change that can be assessed in a statistically significant manner?
Establishing the system - system controls
1/17/13 4:13 PM Experimental Design Harvard Medical
School
72
Additional system controls - eliminating independent variables
A) Limiting Additional Variables that could effect the measurement
example: “What is the effect of caffeine on blood pressure?”
control for: hypertension, other sources of caffeine besides those in the experiment, other chemical perturbations of blood pressure, stress
(What is the effect of caffeine on blood pressure, in the absence of other known pressors?) (Is that what you meant to ask?)
B) What is the effect of Insulin on Akt phoshorylation?
control for: other stimulators of Akt (starve down the cells)
(What is the effect of Insulin in comparison to no growth factors stimulation on Akt phosphorylation) (Is that what you meant to ask?)
1/17/13 4:13 PM Experimental Design Harvard Medical
School
73
Definitions; it is all about language
Definitions are a “big” issue...
What do you call a significant increase in blood pressure?
What do you chacterize as Akt phosphorylation?
Issue of positive controls to set ranges.
Issue of physiologic responses to establish “relevant changes”
1/17/13 4:13 PM Experimental Design Harvard Medical
School
74
Establishing the system in advance of the experiment definitions, time courses and
experimental repeats Does caffeine cause an increase in blood pressure?
A) Definition:
i.What is meant by “caffeine”? (does coffee = caffeine? if not, then one would need to control for the non-caffeine elements of coffe)
ii. What is meant by an “increase” in blood pressure? (degree of difference accepted) iii. What is meant by “blood pressure”? (systolic or diastolic or both)
EXERCISE: note how different an experimental outcome might be, depending on these definitions
1/17/13 4:13 PM Experimental Design Harvard Medical
School
75
Establishing the system in advance of the experiment definitions, time courses and experimental repeats
Does caffeine cause an increase in blood pressure?
B) Time Course
i. Establishes the “usual case”
(example: What color is the sky? What is the effect of caffeine on blood pressure? What is the effect of IGF1 on Akt signaling?
ii. Increases the likelihood of finding more than one answer to the question.
iii. Causes built-in repeats of measurements, but does not count as a “repeat”
1/17/13 4:13 PM Experimental Design Harvard Medical
School
76
Does caffeine cause an increase in blood pressure?
B) Experimental repeats
i. Establishes the stability of the system
ii. Allows for verification or falsification of models
iii. Increases the likelihood to find multiple answers to a question?
(example: What time does the sun rise? What is the effect of caffeine on blood pressure? What is the effect of IGF1 on Akt signaling?
ii. Increases the likelihood of finding more than one answer to the question.
iii. Counts as a repeat!
Establishing the system in advance of the experiment definitions, time courses and experimental repeats
1/17/13 4:13 PM Experimental Design Harvard Medical
School
77
Criteria for Validation - Being able to predict the future
Basis for all clinical testing - “success paramaters” in advance of trials. Meeting success criteria.
P value; probability of random occurence - verification via repetition
Statistics of causal inference
Establishing the system in advance of the experiment
1/17/13 4:13 PM Experimental Design Harvard Medical
School
78
Criteria for Validation - Examples Relevance of Yeast to Humans? (particular examples) Relevance of tissue culture to perturbations in organisms Relevance of subpopulation of humans to broad population of humans
(gender, race)
Validating the System as Being Relevant to Your Issue
1/17/13 4:13 PM Experimental Design Harvard Medical
School
79
Framework: What genes mediate aging? What genes when perturbed, can modulate aging?
Definitional issues: Aging? Positive or negative perturbation?
System choice issues:
1) Short lived organism - bacteria vs yeast vs worms vs mice?
2) Can you do genetics? Can you measure aging? 3) Markers vs phenotypes
4) Validation in other organisms
Examples of going from Framework Question to System, setting up the Experiment
1/17/13 4:13 PM Experimental Design Harvard Medical
School
80
Framework: How is the cell cycle regulated?
Definitional issues: Entire cycle? Particular parts? Positive or negative perturbation? Type of cell?
New Framework: How is G0 regulated in NIH 3T3 cells?
System choice issues:
1) Tissue culture vs general case
2) Genetics & biochemistry? 3) Markers vs phenotypes
4) Validation in whole organisms
Examples of going from Framework Question to System, setting up the Experiment
1/17/13 4:13 PM Experimental Design Harvard Medical
School
81
Framework: Does IGF-1 cause Akt phosphorylation?
Definitional issues: What Akt site? For how long? How much IGF-1? Under what what conditions (Starved? differentiated cells?)
System issues:
1) Tissue culture ? What cell? Rationale for that choice?
2) What antibody? Capability vs Specificity vs Sensitivity Negative & Positive controls for each
3) How do you quantify?
4) Akt phos/Akt total
5) Akt1 vs Akt2 vs Akt3
Examples of going from Framework Question to System, setting up the Experiment
1/17/13 4:13 PM Experimental Design Harvard Medical
School
82
Framework: How does PTEN knockdown cause glioblastoma? What are the genetic changes induced by PTEN knockdown in glial cells, and which are sufficient to induce glioblastoma?
Definitional issues: Degree of genetic changes? Single vs multiple? Time of analysis?
System issues:
1) Tissue culture vs brain? 2) How to knockdown the PTEN? 3) Measuring RNA differences 4) Validating RNA differences 5) Setting up the follow-up systems…
Examples of going from Framework Question to System, setting up the Experiment
1/17/13 4:13 PM Experimental Design Harvard Medical
School
83
Rethinking System Validation – Understanding each perturbation relevant for your hypothesis/question
1. Equipment Validation - Tubes, Heaters, others 2. Reagent Validation - Enzymes, Antibodies,
Mice (knockout vs wt) (Strain Issue) 3. Approach Validation - Biochemical vs Genetic 4. Reductionism Validation - Test tube vs Animal Model vs Human
Expansive View of Systems – Each perturbation
1/17/13 4:13 PM Experimental Design Harvard Medical
School
84
Boot Camp
Introduction to Experimental Design
Class # 3 – Negative and Positive Controls
1/17/13 4:13 PM Experimental Design Harvard Medical
School
85
Designing the Experiment - Negative & Positive Controls
Experimental Program: Includes Distinct Experiments, performed using particular “Controls”
Project B
Experiment 2
FRAMEWORK
Experiment 1
Experiment 2
Project A
Experiment 2
FRAMEWORK
Sy
Experiment 1
System
Experiment 1
System
1/17/13 4:13 PM Experimental Design Harvard Medical
School
87
The Experiment - The Measurement (establishing the measurement is part of establishing the system)
What is the effect of X on Y?
Need to be able to measure “Y” and “not Y”
Need to establish that effect X actually happened.
Therefore if an experiment is a measurement of changes in Y caused by X, you need to be able to prove that:
1) the change in Y can be measured 2) that X happened 3) that any measured change is actually do to X
1/17/13 4:13 PM Experimental Design Harvard Medical
School
88
The Experiment - The Definition of the Negative Control
The negative control can be defined as “The unperturbed by X” control, where X is the thing who’s effect you seek to measure.
The negative control demonstrates that in the experiment “What is effect of X on Y” you have sufficient data to measure the unique effect of “X”. Thus the negative controls are the measurements you need to prove you can measure “X” vs the unperturbed case, AND against everything else perturbed in your system besides “X”; thus the negative control is more than simply “unperturbed”; it’s controlling for everything perturbed in your system perturbed in addition to X, where X is the particular thing you’re interested in.
The Measurement in the setting of an experiment usually means a comparison: seeing a difference between a perturbation and a series of “negative controls.”
1/17/13 4:13 PM Experimental Design Harvard Medical
School
89
The Experiment - The Definition of the Positive Control
The positive control is an internal system validation control.
The positive control demonstrates that in the experiment framed by “What is effect of X on Y?” you have sufficient data to determine that a measurement of “Y” can be determined, and that X is operational.
The positive control is an internal system control – it proves the system is operation in that experiment.
1/17/13 4:13 PM Experimental Design Harvard Medical
School
90
The combination of negative and positive controls allows these determinations:
1. Sensitivity - If X activates Y to degree n, can the system measure degree n?
2. potency - If X activates Y, how much X is necessary to see Y?
3. Degree pf activation - For how long does Y stay activated?
4. relative degree - if X activates Y , but much less well than the positive control , does that alter the interpretation?
5. type - Is there more than one way to activate Y?
6. context - Does X activate Y in a particular cell? Does X activate Y only under a particular condition?
1/17/13 4:13 PM Experimental Design Harvard Medical
School
91
Issues in Measurement - Negative and Positive Controls need to be in effect for these issues
1. Sensitivity - Signal to Noise; can you measure an effect over the baseline signal
2. Potency - Can you determine the maximum effect?
3. Duration - Does the negative stay negative over this period of time? Is the system operational for the required duration?
4. Context - You need distinct negative and positive controls for different ways to activate Y, and for X existing
in different contexts (e.g. cell types)
1/17/13 Experimental Design Harvard Medical
School
92
Negative and Positive Controls for this vector, assuing You are running v-Ras off the CMV?
1/17/13 4:13 PM Experimental Design Harvard Medical
School
93
The Experiment- The Negative Control; not just the unperturbed case, but also the “all but X” control
The negative control as more than the unperturbed case
Unperturbed by What?
The scientist must isolate distinct relevant parts of the Perturbation. You want to determine the effect of v-Ras transformation on NIH3T3 cells
Plasmid expressing CMV-v-Ras-IRES-EGFP + Neo run off a distinct promoter
Therefore you need to keep doing additional negative controls until you have tested all the variables except “X”, where “X” is the thing you are querying in your experiment.
1/17/13 4:13 PM Experimental Design Harvard Medical
School
94
The Experiment- The Positive Control; the internal system control
Experiment 1) WT mouse on a running wheel (negative control for deleting X) 2) X-/- mouse on a running wheel measure exercise capacity and coincident gene and biochemical changes
to understand mechanism. how do you know changes are caused by deletion of X vs running vs
both? 3) X-/- mouse without running (negative control for running) 4) WT mouse without running
how do you know the system is working? that the animal is capable of being perturbed? this is why you need positive controls
5) WT mouse + Known Exercise Enhancer (Testosterone or
Clenbuterol) on a running wheel (need to establish this FIRST)
1/17/13 4:13 PM Experimental Design Harvard Medical
School
95
The Experiment- The Negative Control; from the unperturbed case To the “all but X” case
The negative control as the all but X control
The scientist must isolate distinct relevant parts of the Perturbation. You want to determine the effect of v-Ras transformation on NIH3T3 cells
Plasmid expressing CMV-v-Ras-IRES-EGFP + Neo run off a distinct promoter
Negative controls?: In Red (-) Neo plasmid + Neomycin Empty plasmid CMV IRES EGFP plasmid CMV c-RAS-IRES EGFP CMV-v-RAS-IRES EGFP
1/17/13 4:13 PM Experimental Design Harvard Medical
School
96
The Experiment- The Negative Control; The All-but-X Controls
Negative controls? (-) Neo plasmid + Neomycin Empty plasmid CMV IRES EGFP plasmid CMV-c-RAS-IRES EGFP Measurement negative controls: Depends on what you measure (for
example, do you have to starve the cells to get a “negative” for a particular readout?)
In other words: you need to have a setting where the cells would be both ‘usually not transformed’ but also ‘transformed if there is an
oncogene’ ... thus the question or hypothesis again governs what your control is and what you measure
1/17/13 4:13 PM Experimental Design Harvard Medical
School
97
The Experiment- The Negative Control: refers to every level of the experiment
There are Negative Controls at multiple levels of the experiment The plasmid controls in the previous example are negative controls for the transfection. But if then one wants to measure something, there may be negative
controls to other aspects of the experiment:. Cell types (-) Neo plasmid + Neomycin Empty plasmid CMV IRES EGFP plasmid CMV-c-RAS-IRES EGFP Transformation paradigm: +/- p53 deletion; Metastasis paradigm: +/- VEGF inhibition, etc...
1/17/13 4:13 PM Experimental Design Harvard Medical
School
98
The Experiment- Figuring out what your negative and positive controls are depends on your question or hypothesis, and what system you are using
to address that question or hypothesis...
Consider this question: What is the effect of deleting gene X on exercise capacity?
Consider this hypothesis: Deletion of gene X is sufficient to increase exercise capacity.
System: wt vs X-/- mouse Perturbation: running wheel You want to measure the amount the animal runs, and determine how
deletion of gene X effects running Readouts: 1) Amount the animals run 2) Changes in gene expression caused by deletion of gene X 3) Biochemical changes caused by deletion of gene X 4) Expression and biochemical changes caused by gene X in the
context of running
1/17/13 4:13 PM Experimental Design Harvard Medical
School
99
The Experiment- The Negative Control; the unperturbed by X
Consider this question: What is the effect of deleting gene X on exercise capacity?
Consider this hypothesis: Deletion of gene X is sufficient to increase exercise capacity.
System: Free running wheel Experiment 1) WT mouse on a running wheel (negative control for deleting X) 2) X-/- mouse on a running wheel measure exercise capacity and coincident gene and biochemical changes
to understand mechanism. how do you know changes are caused by deletion of X vs running vs
both? 3) X-/- mouse without running (negative control for running) 4) WT mouse without running
1/17/13 4:13 PM Experimental Design Harvard Medical
School
100
The Experiment- The Negative Control; the unperturbed by X
Consider this question: What is the effect of deleting gene X on exercise capacity?
Consider this hypothesis: Deletion of gene X is sufficient to increase exercise capacity.
System: Free running wheel Experiment 1) WT mouse on a running wheel (negative control for deleting X) 2) X-/- mouse on a running wheel measure exercise capacity and coincident gene and biochemical changes
to understand mechanism. how do you know changes are caused by deletion of X vs running vs
both? 3) X-/- mouse without running (negative control for running) 4) WT mouse without running how do you know changes are caused by deletion of X in a specific
setting? 5) conditional knockouts – tissue specific; time/condition specific
1/17/13 4:13 PM Experimental Design Harvard Medical
School
101
The Experiment- The Positive Control; the internal system control
Experiment 1) WT mouse on a running wheel (negative control for deleting X) 2) X-/- mouse on a running wheel measure exercise capacity and coincident gene and biochemical changes
to understand mechanism. how do you know changes are caused by deletion of X vs running vs
both? 3) X-/- mouse without running (negative control for running) 4) WT mouse without running
how do you know the system is working? that the animal is capable of being perturbed? this is why you need positive controls
5) WT mouse + Known Exercise Enhancer (Testosterone or
Clenbuterol) on a running wheel (need to establish this FIRST)
1/17/13 4:13 PM Experimental Design Harvard Medical
School
102
The Experiment- The Positive Control; the internal system control
WT mouse + Known Exercise Enhancer (Testosterone or Clenbuterol) on
a running wheel (need to establish this FIRST)
It’s only by mapping out this experiment that you see the need to first determine the conditions by which a “known” enhancer of exerciser functions. There are risks to this: you need to make sure your positive control is truly validated... the choice of a positive control and how it is used is critical to the success or failure of an experiment. Using a positive control, you learn how long the experiment needs to run,
when you need to make measurements... what degree of perturbation(s) are necessary...
1/17/13 4:13 PM Experimental Design Harvard Medical
School
103
The Experiment- The Negative Control for a drug trial in the setting of a Genetic mutation
The negative control as the unperturbed case
Unperturbed by What?
The scientist must isolate distinct relevant parts of the Perturbation. Consider this project: Give mice a particular drug - test resistance to a
particular tumor Question: Does tamoxifen decrease tumor load caused by BRCA1 (+)
breast cancer Hypothesis: Tamoxifen decreases tumor load caused by BRCA1 (+)
breast cancer (can you suggest the unperturbed by X controls for this experiment?)
1/17/13 4:13 PM Experimental Design Harvard Medical
School
104
The Experiment- The Negative Control; the unperturbed case
Consider this project: Give mice a particular drug - test resistance to a
particular tumor Question: Does tamoxifen decrease tumor load caused by BRCA1 (+)
breast cancer Hypothesis: Tamoxifen decreases tumor load caused by BRCA1 (+)
breast cancer System: need to induce tumor with BRCA1 (+). Need to see tumor
progression over some time course. A positive control for inhibition would be quite helpful (?deletion of BRCA1?)
Negative controls:
1/17/13 4:13 PM Experimental Design Harvard Medical
School
105
The Experiment- The Negative Control; the unperturbed case
Consider this project: Give mice a particular drug - test efficacy on a
particular tumor Question: Does tamoxifen increase survival in mice with a BRCA1 (+)
breast cancer Hypothesis: Tamoxifen increases survival in mice with a BRCA1 (+)
breast cancer System: need to induce tumor with BRCA1 (+). Need to see tumor
progression over some time course. A positive control for BRCA1 requirement would be quite helpful (deletion of BRCA1?); need to have markers for cell death in the tumor ( why?)
1/17/13 4:13 PM Experimental Design Harvard Medical
School
106
The Experiment- The Negative Control: the all but X controls The Positive Control: the internal system control
Hypothesis: Tamoxifen increases survival in mice with BRCA1 (+) breast
cancer Readout: survival; markers of cell death in the tumor; gene/biochemical
changes induced by tumor vs those perturbed by tamoxifen Experiment Mice (number used determined by prior experiments – need enough to
give statistical significance for tamoxifen effect) MIce, unperturbed (negative control) MIce+ BRCA1 (+) tumor MIce+ Tamoxifen (negative control for tumor) MIce+ BRCA1 tumor + Tamoxifen (test case) positive controls:
1/17/13 4:13 PM Experimental Design Harvard Medical
School
107
The Experiment- The Negative Control: the all but X controls The Positive Control: the internal system control
Mouse, unperturbed Mouse + BRCA1 Mouse + Tamoxifen Mouse + BRCA1 + Tamoxifen are you done? How do you know you gave enough tamoxifen? System design! System controls for sufficient tamoxifen.. You need to know how tamoxifen works, and therefore have a “marker” of
sufficient tamoxifen treatment - you need to give enough tamoxifen so as to perturb that marker (as a positive control to prove the tamoxifen is working)
1/17/13 4:13 PM Experimental Design Harvard Medical
School
108
The Experiment- The Negative Control: the all but X controls The Positive Control: the internal system control
Mouse, unperturbed Mouse + BRCA1 Mouse + Tamoxifen dose 100X Mouse + BRCA1 + Tamoxifen dose X etc (using markers to prove tamoxifen is operative, you establish concentration X... sufficient to
perturb the markers) Mouse + BRCA1 + Tamoxifen dose 2X Mouse + BRCA1 + Tamoxifen dose 10X Mouse + BRCA1 + Tamoxifen dose 100X (why do you do the dose response? establishes the range of efficacy for tamoxifen... the generalizability of claims made as to utility of
the drug)
1/17/13 4:13 PM Experimental Design Harvard Medical
School
109
Next example - caffeine
The negative control as the unperturbed case
What is the effect of caffeine on blood pressure? Experiment: Give 50 people caffeine; measure their blood pressure
at different time points...
This measurement is meaningless unless there is a point of comparison.
That point is the negative control, the unperturbed case.
Experiment: Compare 50 people on nothing to
50 people on caffeine vs One group of 50 people, make sure they don’t have caffeine
for 24 hours, measure their blood pressure, Then give them caffeine, then measure their blood pressure... (the same group can be their own negative control)
The Negative control first establishes a point of reference.
1/17/13 4:13 PM Experimental Design Harvard Medical
School
110
The Experiment- The Negative Control: the all but X controls The Positive Control: the internal system control
Negative control as all-but-X controls:
What is the effect of caffeine on blood pressure? Experiment:
One group of 50 people, make sure they don’t have caffeine for 24 hours, measure their blood pressure, Then give them caffeine, then measure their blood pressure...
objection: is the unperturbed case enough?
1/17/13 4:13 PM Experimental Design Harvard Medical
School
111
2. The negative control as the “all-but X” treatment
What is the effect of caffeine on blood pressure?
A) Give 50 people caffeine in pill form B) Give 50 different people pills that are identical to A) except without caffeine (controls for the caffeine) C) Give 50 different people nothing (controls for the pill) - also controls for the placebo effect in B… determines if there is a placebo effect D) Give 50 people known perturber of blood pressure (positive control)
(alternate design; this design controls for differences between people) A) Give 50 people nothing (establish their baseline) B) Give same 50 people placebo (negative control for caffeine) C) Give same 50 people caffeine (how much? need positive control for caffeine – some marker demonstrating it is in the system) D) Give same 50 people known perturber of blood pressure (positive control for being able to alter blood pressure)
The Experiment- Caffeine experiment The Negative Control; “all but X”; the positive control as an internal system control
1/17/13 4:13 PM Experimental Design Harvard Medical
School
112
Control for surrogates of your desired perturbation
Experimental question: What is the effect of caffeine on blood pressure? Experimental hypothesis: Caffeine injestion causes an increase in blood
pressure Experiment restatement, to allow for the use of coffee as caffeine treatment:
Does caffeinated coffee affect blood pressure, and if so, is the caffeine responsible?
A. No treatment (establishes baseline) B. Water, matched to the amount of coffee - (control for volume) C. Decaffeinated coffee – (all but X control for caffeine_ D. Caffeinated coffee - treatment E. Furosemide – positive control for altering blood pressure
* Eliminate other known blood pressure stimulants or depressants. (to establish baseline... unperturbed by other blood pressure modulators)
1/17/13 4:13 PM Experimental Design Harvard Medical
School
113
Example – genetic pertubation
Experimental question: What is the effect of Ras on proliferation? Experimental hypothesis: Activation of Ras induces cell proliferation. Experiment: Transfect cells with plasmid expressing tamoxifen-inducible
c.a.Ras-ERt2, determine if activation perturbs cell proliferation rates.
Definitions, System controls need to be established...(measuring
transformation) Experiment A. Untransfected cells B. Untransfected cells (+) tamoxifen C. Cells transfected with plasmid containing c.a.Ras-ERt2, (-) tamoxifen D. Cells transfected with plasmid containing c.a.Ras-ERt2, (+) tamoxifen E. Cells transfected with plasmid alone, (-) tamoxifen F. Cells transfected with plasmid alone, (+) tamoxifen G. Cells transfected with gene known to perturb cell proliferation
1/17/13 4:13 PM Experimental Design Harvard Medical
School
114
Additional Controls
Negative Controls: All-but-X controls Positive Controls: Internal System Controls Experimentalist Controls: Part of the system; change who does the
experiment
Blinded analysis!
Pre-existing criteria for evaluation Method Controls: Do the experiment a different way Meta-system controls: Use an entirely different system to address the
framework question Assumption controls: If X leads to Y, and then implies Z, test for Z
Question 1: Is IGF-IR and different isoforms of IRS proteins degraded and ubiquitinated in response to IGF-I, DEX and a cytokine cocktail treatment?
cells + IGF-I
immunoprecipitate IGF-IR, do western blot to check whether IGF-IR protein level decreased (1) and ubiquitin content increased (2)
Repeat the treatment along with proteosomal inhibitor MG132 or lactacystin If (1) yes
and (2) no
Repeat the treatment with cells transiently transfected with Myc-Ub cDNA
IGF-IR is not ubiquitinated and degraded by proteosome after IGF-I treatment
do time- couse assay and use different dosage of IGF-I
both are no both are yes
If (1) yes and (2) no
If (1) yes
(2) no
IGF-IR is ubiquitinated and degraded after IGF-I treatment
IGF-IR is not degraded after IGF-I treatment in muscle cells
Add lysosomal inhibitor(chloroquine) with IGF-I. If IGF-IR protein level remains constant, then IGF-IR is degraded by lysosome.
Repeat the flow chart for + Dex and other atrophy stimulus such as + TNFα, IL-1, inteferon- γ cocktail;Repeat the flow chart for Insulin Receptor and the four isoforms of IRS
Question 2: Which ligases are responsible for the ubiquitination of IGF-IR and IRS cells + IGF-I
+ siRNA to Nedd4 Immunoprecipitate IGF-IR and check whether IGF-IR ubiquitin content decreased
Nedd4 can ubiquitinate IGF-IR
If yes
+ IGF-I + siRNA to Mdm2
Mdm2 can ubiquitinate IGF-IR
If yes
+ IGF-I + siRNA to SOCS1, 2, 3, 4, 5, 6, 7, CIS
SOCS can ubiquitinate IRS
+ IGF-I + siRNA to Mdm2
Mdm2 can ubiquitinate IRS
+ IGF-I + siRNA to c-Cbl and Cbl-b
Cbl can ubiquitinate IRS
Repeat the flow chart for + DEX and + TNFα, IL-1, inteferon- γ cocktail If all are no Transfect cells with siRNA library towards
ubiquitin ligase from Dharmacon Find new E3 ligases responsible for the ubiquitination of IGF-IR and IRS in response to different signals
Incubate cells with crosslinker Immunoprecipitate IRS and do mass-spectrometry to find new IRS interacting proteins
Immunoprecipitate different isoforms of IRS and immunoblot wit hanti-ubiquitin antibody to check whether their ubiquitin content decreased
If yes
If yes
If yes
Fractionate cell homogenate and test the ligase activity in every fraction
Summary 1. Framework. What is the question that you want to answer? 2. Inductive Space. Do you know what you need to know about the subject? Decide what aspects of prior
knowledge relate to your question. Read the literature. 3. System. What tools are you going to use to answer your question? 4. System controls. How do you know your system works? How do you know your system can provide the type
of data you require? Is the system best matched to your question, or might a different system be better? 5. Experiment. What are you going to do to answer the question? Make sure measurements are made multiple
times. Make sure you are measuring the effect in a representative fashion. Study the effect over time, and over a range of the experimental agent. Consult a statistician and discuss the mode of analysis for your data, and how many data points are required.
6. Establish Criteria for the effect, in advance of the experiment. 7. Negative Controls. What negative controls are required? Is there an “all but X” control available? 8. System Positive Controls. How do you know that the system is still operational? What positive controls are
necessary to prove that the thing you want to measure was actually measurable within the context of the experiment?
9. Effect Positive Controls. How do you know that the effect you want to measure can be produced in your system? What positive controls are necessary to produce the effect?
10. Assumption Controls. If X is being measured, can something else be measured that “should” occur when X occurs? If you think Y has happened as a result of X, is there something else you can measure which should also happen when Y happens?
11. Do the experiment 12. Experimentalist Controls. Analyze the data in a blinded fashion 13. Repetition. Repeat the experiment using the same criteria and methodology 14. Model building. What is the answer to the question? 15. Model check. Is the answer responsive to the question? 16. Does the model predict what will happen again? Repeat the experiment. 17. Extension. Does the model hold in different circumstances? 18. Change the system. Approach the question in another way. 19. Change the scientist. See if others can reproduce the effect.